AD- A 174  294 


v. 


:ni 


Vi'l 

‘♦•1 


.» 1 


'  u* 


☆ 

* 


OT' 


t'O 


ASSESSMENT  OF  THE  TECHNICAL  TRAINING 
RECEIVED  BY 

SOURCE  SELECTION  PARTICIPANTS 
IN  AIR  FORCE  SYSTEMS  COMMAND 


THESIS 


Charles  E.  Roberts 
Captain,  USAF 


AFIT/GSM/LSY / 8  6  S - 1 6 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY 


electe 

HQV  2  5  1986 


AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 


Wright-Pafterson  Air  Force  Base,  Ohio 


88  11  25.001 


Sv-1 


AFIT/ GSM/LSY/86 


ASSESSMENT  OF  THE  TECHNICAL  TRAINING 
RECEIVED  BY 

SOURCE  SELECTION  PARTICIPANTS 
IN  AIR  FORCE  SYSTEMS  COMMAND 

THESIS 

Charles  E.  Roberts 
Captain,  USAF 

AFIT/GSM/LSY/86S-16 


DTIC 

ELECTE 

MOV  2  5  1986 


D 


Approved  for  public  release;  distribution  unlimited 


The  contents  of  the  document  are  technically  accurate,  and  no 
sensitive  items,  detrimental  ideas,  or  deleterious  information  is 
contained  therein.  Furthermore,  the  views  expressed  in  the 
document  are  those  of  the  author  and  do  not  necessarily  reflect 
the  views  of  the  School  of  Systems  and  Logistics,  the  Air 
University,  the  United  States  Air  Force,  or  the  Department  of 
Defense. 


AFIT/GSM/LSY/86S-16 


ASSESSMENT  OF  THE  TECHNICAL  TRAINING  RECEIVED  BY  SOURCE 
SELECTION  PARTICIPANTS  IN  AIR  FORCE  SYSTEMS  COMMAND 


THESIS 


Presented  to  the  Faculty  of  the  School  of  Systems  and 

Logistics 

of  the  Air  Force  Institute  of  Technology 
Air  University 

In  Partial  Fulfillment  of  the 
Requirements  for  the  Degree  of 
Master  of  Science  in  Systems  Management 


Charles  E.  Roberts,  B.S. 

Captain,  USAF 

September  1986 

Approved  for  public  release;  distribution  unlimited 


/// 


Acknowl edgement 


I  would  like  to  express  my  sincere  appreciation  to 
Major  Ron  Hitzelberger  for  his  patience  and  support 
throughout  this  thesis  endeavor.  In  addition,  many  thanks 
to  Captain  Steve  Babcock  for  his  assistance  and  insights  as 
we  struggled  through  this  project.  I  would  also  like  to 
*  thank  my  wife  Vicki  for  her  love  and  understanding  during  my 
many  hours  at  the  keyboard.  I  also  want  to  wish  her  the 
best  in  her  future  academic  endeavors. 

Charles  E.  Roberts 


11 


Table  of  Contents 


Page 

Acknowledgement  .  iii 

List  of  Figures . vii 

List  of  Tables  . viii 

Abstract  .  x 

I.  The  Research  Problem  . 1-1 

Introduction  .  1-1 

Background  . 1-2 

Problem  Statement  .  1-3 

Research  Objectives  .  1-4 

Scope  and  Limitations . 1-4 

II.  Literature  Review  .  2-1 

Introduction  .  2-1 

The  Conventional  Approach  .  2-2 

The  Four-Step  Approach  .  2-3 

Source  Selection  Plan  Preparation  .  2-4 

Conclusions  of  Gray  and  Hugo . 2-5 

Technical  Evaluation  of  a  Contract  Proposal .  2-7 

Color  Coding  . 2-8 

III.  Methodology . 3-1 

Introduction  .  3-1 


Phase  I  -  Identify  Survey  Respondents  .  .  .  3-1 

Phase  II  -  Availability  of  Training  ....  3-2 

Phase  III  -  Mail  Survey  Questionnaire  .  .  .  3-3 

Survey  Construction  and  Mailing  ....  3-4 


Administering  the  Survey  .  3-5 

Analysis  of  Results . 3-5 

IV.  Findings  and  Analysis  . . 4-1 

Introduction  .  4-1 


Phase  I  -  Identify  Survey  Respondents  .  .  .  4-1 

Phase  II  -  Availability  of  Training  ....  4-1 

Phase  III  -  Mail  Survey  Questionnaire  .  .  .  4-2 

Step  One  -  Survey  Construction  ....  4-2 

Step  Two  -  Survey  Approval . 4-2 

Step  Three  -  Survey  Administration  .  .  4-3 

Step  Four  -  Analysis  of  Results  ....  4-3 

Response  Rate  and  Background . 4-4 

Response  Rate . 4-4 

iii 


Background  Information  .  4-5 

Research  Objective  1  .  4-9 

Survey  Questions  and  Statistical 

Techniques . 4-9 

Findings  and  Analysis  .  4-9 

Research  Objective  2  .  4-12 

Findings  and  Analysis  .  4-12 

Research  Objective  3  .  4-14 

Survey  Questions  and  Statistical 

Techniques . 4-14 

Findings  and  Analysis  .  4-14 

Research  Objective  4  .  4-18 

Survey  Questions  and  Statistical 

Techniques . 4-18 

Findings  and  Analysis  .  4-18 

Research  Objective  5  .  4-22 

Findings  and  Analysis  .  4-22 

Research  Objective  6  .  4-23 

Survey  Questions  and  Statistical 

Techniques . 4-2  3 

Findings  and  Analysis  .  4-23 

Research  Objective  7  .  4-25 

Survey  Questions  and  Statistical 

Techniques . 4-2  5 

Findings  and  Analysis  .  4-25 

V.  Conclusions  and  Recommendations  .  5-1 

Chapter  Overview  .  5-1 

Preparedness  of  Source  Selection 

Participants  .  5-1 

Conclusions . 5-1 

Discussion . 5-2 

Recommendations  .  5-3 

Benefits  of  a  Source  Selection 

Training  Course  .  5-4 

Conclusions . 5-4 

Discussion . 5-4 

Recommendations  .  5-5 

Format,  Content,  and  Length  of  a 

Training  Course  .  5-5 

Conclusions . 5-5 

Discussion . 5-6 

Recommendations  .  5-7 

Area  For  Additional  Research . 5-8 

Appendix  A:  Survey  Cover  Letter  .  A-l 

Appendix  B:  Survey  Instrument  .  B-l 

Appendix  C:  Comments  and  Responses  to  Survey  Questions  C-l 


Bibliography  .  Bib-1 

Vita  . V-l 


List  of  Fiqures 


Figure 

1.1  Source  Selection  Organization 


Page 


List  of  Tables 


Table  Page 

4.1  Predict  Division  Fesponse  Rate  .  4-4 

4.2  Military  Rank  or  Civilian  Grade  of  Survey 

Respondents . 4-5 

4.3  Phase  of  the  Acquisition  Process  the  Respondents 

Were  Most  Familiar  With . 4-6 

4.4  Type  of  Acquisition  Respondents  Were  Most 

Familiar  With . 4-6 

4.5  Current  Functional  Area  of  Survey  Respondents  .  4-7 

4.6  Education  Level  of  Survey  Respondents  .  4-7 

4.7  Years  of  Soiree  Selection  Experience  .  4-8 

4.8  Number  of  Source  Selections  Survey  Respondents 

Have  Participated  In . 4-8 

4.9  *  Number  of  Technical  Panels  Survey  Respondents 

Have  Been  a  Member  Of . 4-9 

4.10  Did  Survey  Respondents  Receive  Source  Selection 
Training  Prior  To  Their  First  Source  Selection?  4-10 

4.11  Did  Source  Selection  Participants  Receive  Any 
Source  Selection  Training  Prior  to  Subsequent 

Source  Selections?  .  4-10 

4.12  From  What  Source  Did  the  Survey  Participants 

Receive  Their  Source  Selection  Knowledge?  .  .  .  4-11 

4.13  Formal  Air  Force  Training  Courses  Containing 

Source  Selection  Material  that  Survey  Respondents 
Have  Attended . 4-12 

4.14  Would  Survey  Respondents  Attend  a  Source  Selection 

Training  Course  if  One  Was  Available? . 4-14 

4.15  Contingency  Table  -  Source  Selection  Experience  by 

Willingness  to  Attend  Source  Selection  Training  4-15 

4.16  How  Beneficial  the  Respondents  Felt  a  Formal 

Course  Would  Be  in  Providing  Information  on  the 
Source  Selection  Process  and  Procedures  ....  4-16 


Table 


Page 


4.17  How  Beneficial  Respondents  Felt  a  Contractor 

Presentation  Would  Be  During  the  Technical 
Evaluation  Process . 4-17 

4.18  How  Beneficial  the  Respondents  Felt  a  Formal 

Course  Would  Be  in  Providing  Fanel  Members  the 
Required  Technical  Background  .  4-18 

4.19  Would  the  Technical  Panel  or  Source  Selection 

Schedule  Allow  for  a  Training  Course  to  be 
Conducted? . 4-15 

4.20  Length  of  the  Source  Selection  Course  .  4-20 

4.21  The  Content  of  the  Source  Selection  Training 

Course  Should  Include  .  4-20 

4.22  Format  for  the  Source  Selection  Training  Course  4-21 

4.23  Source  Selection  Training  Course  Should  Be 

Given  By . 4-21 

« 

4.24  How  Individuals  Were  Selected  for  a  Technical 

Panel . 4-24 

4.25  How  the  Survey  Respondents  Select  Individuals  for 

Membership  on  a  Technical  Panel  .  4-24 

4.2  6  How  the  Respondents  Felt  Technical  Panel  Members 

Should  Be  Selected . 4-25 

4.27  Did  Formal  Training  Adequately  Prepare 

Respondents  for  Source  Selection?  .  4-26 


4.28  How  Well  Prepared  Were  the  Respondents  for  Their 

Responsibilities  or  the  Technical  Panel?  ....  4-27 

4.29  How  Prepared  the  Respondents  Felt  Other  Members 
of  the  Technical  Panel  Were  for  Their 

Responsibilities  .  4-28 

4.30  The  Biggest  Problem  Encountered  as  a  Source 

Selection  Member . 4-29 

4.31  What  the  Respondents  Felt  Were  the  Major  Problem 
Areas  in  the  Evaluation  of  Contractors  ' 

Technical  Proposals  .  4-29 

4.32  Contingency  Table  -  Experience  by  Technical  Panel 

Preparedness  .  4-30 


viii 


AFIT/GSM/LSY/86S-1 6 


Abstract 


Source  selection  is  an  important  and  complex  part  of 
the  Air  Force  acquisition  process  and  requires  the  skills 
and  talents  of  many  Air  Force  Systems  Command  (AFSC) 
personnel.  This  research  assessed  the  preparedness  of  AFSC 
source  selection  participants.  In  particular,  technical 
panel  members  were  surveyed  to  determine  how  prepared  they 
were  for  their  participation  in  the  technical  evaluation  of 
contract  proposals.  The  key  issues  addressed  were  the 
preparedness  of  the  source  selection  participants  and  the 
availability  and  benefit  of  source  selection  training. 

This  investigation  was  accomplished  by  sending  a  survey 
questionnaire  to  source  selection  participants  in  the  six 
Air  Force  Systems  Command  product  divisions.  The  results 
show  that  75  percent  of  the  technical  panel  members  had  not 
received  any  formal  source  selection  training  prior  to  their 
first  source  selection.  The  majority  of  the  technical  panel 
members  felt  technically  qualified  to  accomplish  the 
technical  evaluation  required  during  the  source  selection. 
However,  they  felt  additional  training  on  general  source 
selection  procedures  would  be  beneficial.  Key  topics 
recommended  for  training  included  instructions  on  the 
application  of  evaluation  criteria  and  standards,  and 
instructions  on  completing  source  selection  forms. 


ASSESSMENT  OF  THE  TECHNICAL  TRAINING  RECEIVED  BY  SOURCE 
SELECTION  PARTICIPANTS  IN  AIR  FORCE  SYSTEMS  COMMAND 


I.  The  Research  Problem 


Introduction 

The  acquisition  of  major  weapons  systems  in  the  United 
States  Air  Force  is  a  very  important  and  complicated 
process,  and  involves  large  expenditures  of  taxpayers' 
money.  Congress,  the  Department  of  Defense,  and  the  tax- 
paying  public  increasingly  scrutinize  the  procurement 
process  to  ensure  the  Government  purchases  quality  products 
at  fair  and  reasonable  prices.  The  process  of  evaluating 
contract  proposals  is  a  critical  step  in  the  Air  Force 
procurement  process.  "There  are  a  number  of  qualified 
sources  in  the  United  States  that  have  prerequisite 
experience,  capabilities,  and  facilities  for  the  development 
and  production  of  major  systems  and  subsystems"  (1:2). 
Because  the  potential  need  exists  for  evaluating  large 
numbers  of  proposals  for  a  given  new  system,  the  Air  Force 
developed  a  process  for  evaluating  proposals  called  Source 
Selection.  Captain  Robert  Gray  and  Captain  Jeffrey  Hugo 
identified  the  lack  of  training  for  source  selection 
participants  in  their  thesis  Weapon  System  Source  Selection : 
An  Assessment  of  Air  Force  Source  Selection  Approaches  as  a 
critical  Air  Force  problem  (7:6-2).  This  research  will 
specifically  identify  the  background  and  training  of  source 
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selection  participants  prior  to  their  participation  in  the 
technical  evaluation  of  a  contract  proposal. 


Background 

"The  principle  objective  of  the  source  selection 
process  is  to  select  the  source  whose  proposal  has  the 
highest  degree  of  credibility  and  whose  performance  can  be 
expected  to  best  meet  the  government's  requirements  at  an 
affordable  cost.  The  source  selection  process  should  be 
properly  structured  to  balance  technical,  cost,  management, 
and  logistic  considerations  consistent  with  the  phase  of 
acquisition,  program  requirements,  and  business  and  legal 
constraints"  (5:3). 

Recent  legislation  has  illustrated  that  Congress  is 
very  concerned  about  the  issue  of  competition  in  government 
procurement.  The  increased  emphasis  on  competition  places 
added  pressure  on  source  selection  participants  to 
thoroughly  evaluate  each  proposal.  According  to  Smail  and 
Simmons : 

With  the  passage  of  the  Competition  in  Contracting 
Act  of  1984  (CICA),  the  Deficit  Reduction  Act  of 
1984,  the  Small  Business  and  Federal  Procurement 
Competition  Enhancement  Act  of  1984,  the  Defense 
Authorization  Act  of  1985,  the  98th  Congress 
clearly  mandated  its  intent  to  increase  competition 
in  government  contracting  by  allowing  fewer 
noncompetitive  acquisitions.  Thus  it  is 
anticipated  that  the  number  of  competitive 
procurements  will  increase  dramatically  in  the 
years  to  come  [15:1]. 

The  intent  of  the  legislation  is  to  ensure  that  all 
potential  contractors  be  given  a  fair  and  equal  opportunity 
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to  compete.  The  objective  of  the  source  selection  process 
is  to  ensure  impartial  and  comprehensive  evaluation  of  a 
contractor's  proposal.  The  purpose  of  the  technical  panel 
is  to  evaluate  the  technical  aspects  of  each  proposal  to 
determine  which  contractor  will  produce  a  system  capable  of 
meeting  Air  Force  requirements. 

Problem  Statement 

The  Gray  and  Hugo  thesis  identified  the  lack  of 
training  received  by  source  selection  participants  as  Deing  . 
a  general  problem.  They  did  not  identify  what  training  was 
available  or  whether  source  selection  participants  were 
aware  of  or  would  attend  any  training  course.  Often  the 
technical  evaluation  of  a  contract  proposal  will  require  a 
high  degree  of  technical  expertise  to  even  understand  the 
proposal.  Then  the  technical  merits  of  each  proposal  must 
be  analyzed.  Therefore,  a  sufficient  level  of  technical 
knowledge  must  be  evident  for  the  evaluation  to  be  complete 
and  thorough.  My  specific  research  question  is:  What 
specific  training  is  required  for  participants  in  the 
technical  evaluation  of  a  contract  proposal?  What  do  people 
with  previous  source  selection  experience  say  is  required  of 
participants  in  the  technical  evaluation  of  a  contract 
proposal?  Is  further  technical  training  required,  or  is 
training  in  general  source  selection  procedures  needed? 

This  research  will  attempt  to  answer  these  questions  by 
conducting  a  survey  of  Source  Selection  Department  Chiefs, 


Scarce  Selection  Officers,  and  people  that  have  participated 
in  previous  and  on-going  source  selections. 

Research  Objectives 

1.  Determine  if  any  general  or  selection-specific  education 
was  received  by  source  selection  participants  prior  to  a 
source  selection. 

2.  Determine  what  courses  currently  exist  that  specifically 
cover  or  include  material  on  the  source  selection  process. 

3.  Determine  if  a  source  selection  training  course  would  be 
beneficial  to  technical  panel  participants  or  source 
selection  participants  in  general. 

4.  Determine  the  format  and  length  of  any  source  selection 
training  course  if  the  need  for  one  is  identified. 

5.  Determine  the  level  of  expertise  required  by  regulation 
for  membership  on  a  technical  panel. 

6.  Determine  how  technical  panel  members  are  chosen. 

7.  Determine  how  well  prepared  members  of  previous 
technical  panels  were  for  their  responsibilities  on  the 
panel . 

Scope  and  Limitations  of  Research 

The  majority  of  the  Air  Force's  acquisition  funds  are 
spent  by  Air  Force  Systems  Command  (AFSC)  Product  Divisions, 
so  the  focus  of  this  research  will  be  on  source  selection 
personnel  and  training  within  AFSC.  Procurements  conducted 
outside  of  AFSC  Product  Divisions  will  not  be  considered. 
During  the  source  selection  process  a  contractor 's 


proposal  is  evaluated  under  four  basic  areas  of 
consideration:  cost,  technical,  management,  and  logistics 

(2:21).  These  four  areas  are  evaluated  by  panels  comprising 
the  Source  Selection  Evaluation  Board  (SSEB).  The  SSEB  is 
responsible  for  the  review  and  evaluation  of  each 
contractor's  proposal  against  the  solicitation  requirements, 
•  the  approved  evaluation  criteria,  and  the  standards.  The 
SSEB  is  typically  organized  as  shown  in  Figure  1.1.  The 
SSEB  identifies  any  proposal  deficiencies  to  the  appropriate 
contractor  and  determines  whether  the  proposals  meet  minimum 
government  standards.  All  recommendations  of  the  SSEB  are 
reviewed  by  the  Source  Selection  Advisory  Council  (SSAC). 

The  SSAC  is  a  group  of  senior  government  personnel  appointed 
by  the  Source  Selection  Authority  (SSA)  to  complete  a 
comparative  analysis  of  the  results  of  the  SSEB.  The  SSA  is 
the  individual  designated  to  direct  the  source  selection  and 
make  the  final  source  selection  decision  (5:4). 

This  research  will  focus  specifically  on  the  training 
requirements  for  participants  in  the  technical  evaluation 
area  of  the  SSEB  (Technical  Panel  members  in  Figure  1.1). 
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SOURCE  SELECTION  AUTHORITY 
( SSA ) 


SOURCE  SELECTION 
(SS 

ADVISORY  COUNCIL 
,AC) 

SOURCE  SELECTION  EVALUATION  BOARD 
( SSEB ) 

I 


TECHNICAL 

PANEL 

COST 

PANEL 

MANAGEMENT 

PANEL 

LOGISTICS 

PANEL 

Figure  1.1.  Source  Selection  Organization. 
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II.  Literature  Review 


Introduction 

The  procurement  of  major  weapons  systems  in  the  United 
States  Air  Force  is  a  very  complex  and  time  consuming 
process.  An  adequate  assessment  of  any  training  requirement 
for  a  particular  phase  of  the  process  requires  a  basic 
understanding  of  the  source  selection  process  in  general. 
Each  procurement  requires  the  development  by  the  Air  Force 
of  the  system's  specifications  and  requirements,  and 
involves  coordination  between  many  different  Air  Force 
agencies.  "Once  the  Air  Force  is  able  to  specify  the 
desired  system  and  define  proposal  evaluation  criteria, 
potential  contractors  are  notified  by  means  of  a  'Request 
for  Proposal'  (RFP)"  (2:39).  The  RFP  is  the  medium  by  which 
a  contractor  is  introduced  to  the  job  desired  by  the 
government.  It  conveys  a  complete  description  of  the  work 
to  be  performed  and  allows  the  government  to  determine  the 
capability  and  price  of  the  contractor's  efforts  (2:39). 

The  Air  Force  uses  the  source  selection  process  to 
evaluate  each  contractor  proposal.  All  proposals  received 
are  considered,  with  the  goal  being  to  determine  the  company 
or  companies  best  able  to  produce  the  system.  The  primary 
objectives  of  the  source  selection  process  are: 

1.  Maximize  competition; 

2.  Minimize  the  complexity  of  the  solicitation; 

3.  Ensure  impartial  and  comprehensive  evaluation  of  an 


offeror's  proposals;  and 


4.  Ensure  the  selection  of  the  source  whose  proposal 
has  the  highest  degree  of  realism  and  whose  performance 
is  expectecd  to  best  meet  stated  Government 
requirements  [18:2]. 

Presently  there  are  a  variety  of  different  approaches 
used  to  evaluate  proposals  during  the  source  selection 
process.  The  two  primary  approaches  are  the  "Conventional" 
approach  and  the  "Four-Step"  approach.  The  following  is  a 
brief  description  of  the  two  approaches. 

The  Conventional  Approach .  The  conventional  approach 
consists  of  three  main  phases.  In  the  first  phase,  the  SSEB 
grades  each  submitted  proposal  against  the  established 
evaluation  criteria.  The  functional  panels  (cost, 
technical,  management,  and  logistics)  perform  an  independent 
evaluation  of  each  proposal  and  the  results  are  summarized 
for  the  SSAC  and  SSA.  In  the  second  phase,  the  "competitive 
range"  is  determined.  The  competitive  range  includes  all 
contractor  proposals  that  fall  within  established  minimum 
and  maximum  requirement  levels.  Only  proposals  that  fall 
within  the  competitive  range  continue  into  parallel 
negotiations.  Negotiation  involves  a  bargaining  process 
between  the  government  and  the  contractor  to  "iron  out" 
terms  of  the  contract  (8:60).  During  the  source  selection 
process  the  SSEB  generates  Deficiency  Reports  (DR)  and 
Contractor  Inquiries  (Cl)  (2:63).  A  DR  identifies  a  part  of 
the  proposal  that  does  not  meet  the  established  minimum 
requirements  or  standards,  and  a  Cl  requests  additional 
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information  or  cl  arif  ication  concerning  a  part  of  the 
proposal.  Once  the  negotiations  are  completed,  the 
contractors  are  instructed  to  submit  their  "best  and  final 
offer"  (14:1-1).  Phase  three  involves  analysis  of  the  final 
contractor  offers  and  selection  of  the  winning  contractor. 
The  SSA  chooses  the  proposal  best  meeting  the  government's 
requirements,  the  winning  contractor  is  notified,  and  the 
contract  is  signed.  All  unsuccessful  contractors  are 
notified  of  the  contract  award. 

The  Four-Step  Approach .  The  Federal  Acquisition 
Regulation  describes  the  Four-Step  Approach  as  follows 
(18:15.613;  19): 

Step  One  -  Evaluation  and  Discussion  of  Technical 
Proposals .  Technical  proposals  are  solicited  from  all 
competing  contractors  and  evaluated  on  how  well  they  satisfy 
government  requirements  just  as  in  the  conventional 
approach.  After  the  evaluations  are  completed,  the 
unqualified  bidders  are  eliminated. 

Step  Two  -  Evaluation  and  Discussion  of  Cost/Price 
Proposals .  After  the  technical  proposals  are  analyzed,  the 
remaining  contractors  are  required  to  submit  cost/price 
proposals.  A  competitive  range  is  established,  and  again  a 
contractor  can  be  eliminated  from  further  consideration  if 
his  proposal  falls  outside  this  competitive  range.  During 
Step  2,  discussions  are  held  with  the  contractors  to  clarify 
portions  of  the  proposal  if  necessary. 

Step  Three  -  Common  Cutoff .  In  step  three,  a 


common  cutoff  date  is  established  for  the  receipt  of  all 
technical  and  cost/price  proposal  revisions.  After  the 
cutoff  date,  all  remaining  proposals  are  evaluated  and  a 
single  contractor  is  selected  to  enter  Step  Four. 

Step  Four  -  Final  Negotiations  and  Contra ct  Award . 
In  Step  Four,  the  government  and  the  selected  contractor 
negotiate  a  final  contract  acceptable  to  both  parties.  When 
an  agreement  is  reached,  the  contract  is  signed  and  the 
unsuccessful  contractors  are  notified. 

The  important  point  is  the  difference  between  the  two 
procedures.  The  "Four-Step"  process  differs  significantly 
from  the  "Conventional"  method  in  that  the  offeror's 
technical  and  cost  proposals  are  not  submitted  and  evaluated 
simultaneously,  definitive  contracts  are  not  negotiated  with 
all  offerors,  and  the  apparent  contractor  is  selected  and 
announced  prior  to  negotiation  of  a  definitive  contract 
(2:53).  "In  the  four  step  process,  proposal  deficiencies 
are  not  revealed  to  the  individual  offerors.  These 
deficiencies  are  disclosed  and  resolved  during  Step  Four, 
Negotiations  of  a  Definitive  Contract"  (2:53). 

Source  Selection  Plan  Preparation 

The  Air  Force  initiates  the  source  selection  process  by 
generating  a  Source  Selection  Plan  (SSP).  The  SSP  is 
prepared  prior  to  the  release  of  the  Request  for  Proposal, 
and  outlines  the  course  of  action  to  be  followed  throughout 
the  source  selection  process.  The  SSP  also  establishes  the 
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standards  and  criteria  to  be  used  in  evaluating  contractor's 
proposals  by  identifying  broad  areas  of  the  program  such  as 
logistics,  management,  operations,  etc.  and  ranking  the 
areas  in  order  of  relative  importance  to  the  program  (2:22). 
The  SSP  controls  the  entire  evaluation  process  and  must  be 
approved  by  the  Source  Selection  Authority  (SSA).  "The  SSA 
is  responsible  for  the  proper  and  efficient  conduct  of  the 
'  entire  source  selection  process  encompassing  proposal  soli¬ 
citation,  evaluation,  selection  and  contract  award"  (5:5). 

Once  the  SSP  is  approved,  the  government  issues  the  RFP 
which  asks  industry  to  submit  contract  proposals.  After  all 
the  proposals  are  received,  the  individual  panels  comprising 
the  Source  Selection  Evaluation  Board  (SSEB)  evaluate  the 
proposals  against  the  previously  established  standards. 

This  is  a  very  important  point;  contractor  proposals  are 
evaluated  against  the  established  standards  and  not  against 
each  other.  After  the  evaluation,  the  SSEB  presents  its 
recommendations  to  the  Source  Selection  Advisory  Council 
( SSAC ) .  The  SSAC  reviews  the  board's  recommendations  and 
then  submits  the  recommendations  to  the  Source  Selection 
Authority  (SSA).  The  SSA  makes  the  final  selection,  and  the 
Contracting  Officer  executes  the  selected  firm's  contract 
for  the  government.  The  SSA  also  notifies  the  unsuccessful 
contractors  that  the  contract  has  been  awarded  (2:19-27). 


Conclusions  of  Gray  and  Hugo 

Captain  Robert  Gray  and  Captain  Jeffrey  Hugo  conducted 
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an  investigation  that  examined  the  different  approaches  used 
in  the  Air  Force  source  selection  process.  Their  research 
surveyed  the  opinions  of  source  selection  personnel  in  an 
effort  to  determine  the  best  approach  for  varying 
acquisition  situations.  Their  results  showed  that  the 
conventional  approach  was  the  overall  preferred  approach. 
Their  results  also  showed  that  the  personnel  surveyed  felt 
the  technical  aspects  of  a  weapons  system  are  adequately 
defined  for  the  contractor  in  the  Request  for  Proposal. 
Finally,  and  most  important  for  this  research,  their  results 
showed  that  89  percent  of  the  source  selection  personnel 
surveyed  did  not  receive  training  prior  to  participating  in 
their  first  source  selection  (7:4-10). 

Gray  and  Hugo  found  that  the  lack  of  source  selection 
training  had  a  definite  impact  on  the  source  selection 
process  (7:6-2).  They  drew  the  following  conclusions: 

1 .  Air  Force  personnel  are  not  receiving  adequate 
training  prior  to  source  selection  participation.  Only 
11  percent  of  the  Air  Force  personnel  surveyed  received 
training  before  participating  in  their  first  source 
selection. 

2.  Formal  Air  Force  training  courses  are  not  covering 
the  source  selection  process  in  enough  detail.  Thirty- 
six  percent  of  the  source  selection  personnel  surveyed 
have  never  attended  formal  training.  Personnel  gain 
the  bulk  of  their  source  selection  experience  from  on- 
the-job  training  and  self  study  [7:6-2]. 

Gray  and  Hugo  recommended  that  a  comprehensive  training 

course  be  established  to  cover  all  aspects  of  the  source 

selection  process.  The  course  design  should  include  source 

selection  plan  preparation,  basic  source  selection  forms. 


Air  Foi'ce  source  selection  philosophy  (7:6-3).  This 
research  will  focus  on  the  specific  training  requirements 
for  the  technical  evaluation  of  contract  proposals. 


Technical  Evaluation  of  a  Contract  Proposal 

Under  the  technical  area  of  evaluation,  the  following 
criteria  are  used  to  evaluate  the  offeror's  proposal: 

1.  Understanding  of  the  technical  aspects  of  the  program. 
The  offeror  must: 

--Demonstrate  an  understanding  of  the  purpose  and  scope 
of  the  program. 

--Define  trade-offs  which  will  influence  program 
decisions . 

— Exhibit  an  understanding  of  technology  availability, 
applicability,  and  limitations. 

2.  Risk  assessment.  The  offeror  must: 

--Identify  and  rank  performance,  cost,  and  schedule 
risks  and  provide  information  on  the  risks  and  problem 
areas . 

--Provide  the  rationale  for  selecting  approaches  with 
risk  and  identify  acceptable  alternatives  to  risk  areas 
showing  the  relationship  of  technology  risks  to  cost  risks. 

--Provide  schedules  for  resolving  risks  on  task 
sequence  and  overall  program  schedules. 

3.  Soundness  of  approach.  The  offeror  must: 

— Summarize  the  proposed  approaches  and  justify  the 
approaches  by  relating  to  tried  and  proven  designs,  methods 


VuSvWa  A.'  .. 


i-Vsf.V  -A -C L-.V.'.' V\ 


or  procedures,  and  by  demonstrating  the  feasibility  and 
payoff  of  any  new  or  innovative  approaches. 

--Demonstrate  that  the  approaches  used  efficiently  integrate 
the  various  disciplines  and  functional  areas  to  meet  program 
objectives  and  that  tasks  flow  in  a  logical  sequence  and  are 
properly  prioritized. 

— Demonstrate  that  adequate  resources  exist  to  accomplish 
the  proposed  effort  (1:29;  17). 


Color  Codinc 


After  assessing  the  offeror's  data,  the  evaluator  will 
apply  the  rating  scheme  prescribed  in  the  SSP  and  rate  each 
proposal  in  relation  to  the  standards.  Different  coding 
schemes  may  be  used  depending  on  the  level  of  evaluation 


during  the  source  selection  (e.g.,  colors,  symbols, 
numbers).  The  standard  color  code  scheme  specified  in  AFR 


70-15  and  used  most  often  is: 


COLOR  DEFINITIONS 

Blue  Exceptional  Exceeds  specified  performance  or 

capability  in  a  beneficial  way  to 
the  Air  Force;  high  probability  of 
success;  no  significant  weakness. 

Green  Acceptable  Meets  standards;  good  probability 

of  success;  weaknesses  can  be 
readily  corrected. 

Yellow  Marginal  Fails  to  meet  standards;  low 

probability  of  success;  significant 
deficiencies  but  correctable. 

Red  Unacceptable  Fails  to  meet  minimum  requirements; 

needs  a  major  revision  to  the 
proposal  to  make  it  correct  [5:12]. 

The  full  range  of  ratings  should  be  used  so  that  the 


variances  between  proposals  may  be  readily  identified. 


Ill .  Methodology 


Introduction 

This  chapter  describes  the  methodology  used  to  complete 
this  research  project.  The  research  was  conducted  in  three 
phases.  The  first  (preliminary)  phase  involved  telephone 
and  personal  interviews.  The  interviews  were  conducted  to 
compile  a  list  of  experienced  source  selection  personnel 
(the  sample  population)  and  to  determine  the  availability  of 
source  selection  training  programs.  The  second  phase 
involved  a  review  of  source  selection  records  and  reports, 
and  a  literature  search  to  determine  what,  if  any,  source 
selection  training  was  available.  In  the  third  phase,  a 
mail  survey  was  used  to  gather  opinion  data  and  demographic 
information  from  respondents  concerning  source  selection 
training.  Each  phase  is  described  in  more  detail  below. 

Phase  1^  -  Identify  Survey  Respondents 

Telephone  and  personal  interviews  were  conducted  with 
Source  Selection  Department  Chiefs  in  Systems  Command 
Product  Divisions  to  compile  a  survey  response  mailing  list. 
An  effort  was  made  to  contact  the  same  respondents  that 
Captain  Gray  and  Captain  Hugo  contacted.  A  personal 
interview  was  conducted  with  Mr.  Jim  Helmig  at  the 
Aeronautical  Systems  Division  and  a  telephone  interview  was 
conducted  with  Ms.  Sheila  Keeling  at  the  Ballistic  Missile 
Office.  The  rest  of  the  product  divisions  were  contacted  by 


mail,  and  asked  to  verify  the  survey  mailing  list  obtained 
from  Captain  Gray  and  Captain  Hugo.  In  addition,  various 
respondents  appearing  on  the  mailing  list  were  contacted  and 
asked  to  provide  the  names  of  other  source  selection 
participants  not  included  in  the  Gray  and  Hugo  survey.  This 
was  done  to  increase  the  sample  size  and  to  ensure  that 
experienced  source  selection  participants  were  not 
overlooked.  It  was  impossible  to  identify  the  entire 
population  of  interest  due  to  time  limitations  and  the 
transitory  nature  of  the  Systems  Command  job  environment. 
Thus,  the  mailing  list  contained  a  representative, 
nonrandom,  sample  of  the  source  selection  population.  The 
nonrandom  (nonprobabilistic)  sample  was  used  for  several 
reasons.  First,  the  entire  population  could  not  be 
identified,  thus  eliminating  the  possibility  of  any  random 
sampling.  Second,  the  information  needed  required  responses 
from  people  experienced  in  the  source  selection  process. 

In  an  effort  to  include  participants  in  an  on-going 
source  selection.  Major  Scott  Graham,  the  Technical  Panel 
Executive  Officer  for  the  Advanced  Tactical  Fighter  ( ATF ) 
Source  Selection  was  contacted.  He  agreed  to  distribute  25 
surveys  to  members  of  the  technical  panel.  Respondents  in 
an  in-progress  source  selection  were  solicited,  to  assess 
the  most  current  training  received  by  the  participants. 

Phase  II  -  Availability  of  Training 

A  literature  search  was  conducted  to  determine  the 
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availability  of  training  programs  covering  source  selection. 
In  addition,  during  the  personal  interviews  the  respondents 
were  questioned  about  the  availability  of  source  selection 
training  courses  in  Systems  Command.  Air  Force  Institute  of 
Technology  (AFIT)  Professional  Continuing  Education  ( PCE ) 
course  curriculums  were  reviewed  to  determine  to  what  extent 
source  selection  topics  were  covered. 

Business  Management  Research  Associates  Inc.  (BMRA)  of 
Arlington  Virginia  under  contract  to  the  Air  Force  Business 
Management  Research  Center  (AFBMRC)  at  Wright-Patterson  AFB 
conducted  a  study  to  determine  the  needs  for  source 
selection  training.  Their  final  report  was  used  to  provide 
background  material  and  information  on  the  availability  of 
source  selection  training  programs.  A  personal  interview 
was  conducted  in  Arlington  Virginia  with  Mr.  John  Lynch  of 
BMRA  to  discuss  the  results  of  their  study.  The  BMRA  study 
determined  what  source  selection  training  programs  were 
currently  available  in  the  Department  of  Defense,  and 
recommended  that  source  selection  training  courses  be 
established  (4). 

Phase  III  -  Mail  Survey  Questionnaire 

A  thirty-nine  question  survey  was  constructed,  tested 
and  sent  to  the  205  Systems  Command  Product  Division 
respondents  identified  during  Phase  I  of  my  research.  The 
responses  were  analyzed  using  the  SPSSx  statistical  package 
on  the  Academic  Support  Computer  at  AFIT. 
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Survey  Construction  and  Mail inq .  A  mail  survey 
questionnaire  was  chosen  as  the  primary  instrument  to  obtain 
source  selection  training  information.  The  mail  survey  was 
chosen  for  a  number  of  reasons: 

1.  The  large  sample  size  precluded  using  another  method 
(such  as  telephone  interviews)  because  of  time  restrictions. 

2.  Personal  interviews  could  not  be  conducted  due  to  the 
diverse  locations  of  Systems  Command  Product  Divisions. 

3.  The  mail  survey  was  the  least  costly  method  in  terms  of 
manpower  and  time. 

4.  "Mail  surveys  are  typically  perceived  as  being  more 
impersonal,  providing  more  anonymity  than  the  other 
communications  modes"  (6:172). 

'The  survey  questionnaire  itself  was  originally 
comprised  of  three  sections,  a  background/demographics 
section,  a  section  containing  general  source  selection 
questions,  and  a  section  containing  questions  for  technical 
panel  participants  (10:132-143).  The  Air  Force  Survey 
Control  Office,  the  approval  authority  for  the  survey, 
required  that  the  survey  be  combined  with  a  similar  survey 
assessing  the  training  requirements  in  the  area  of  cost. 
Therefore,  the  final  survey  contained  four  sections,  a 
background/demographics  section,  a  section  with  general 
source  selection  questions,  a  section  with  questions  for 
technical  panel  members,  and  a  section  with  questions  for 
cost  panel  members.  The  approved  survey  is  included  in 
Appendix  B  to  this  report.  The  questions  used  in  the  survey 
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were  designed  to  assess  the  opinions  and  preferences  a' 
experienced  source  selection  personnel  on  the  topic  of 
training  for  source  selection  participants.  The 
questionnaire  was  tested  prior  to  approval  in  an  effort  to 
improve  the  quality  of  the  survey  instrument.  Several 
students  in  the  AFIT  Graduate  Systems  Management  program 
completed  the  survey  and  provided  comments  and  suggestions 
for  improvement. 

Administering  the  Survey .  Once  the  Air  Force  Survey 
Control  Office  at  the  Air  Force  Military  Personnel  Center 
( AFMPC )  approved  the  survey,  a  survey  packet  was  prepared 
for  each  person  on  the  mailing  list.  Each  survey  packet 
contained  a  cover  letter,  survey,  and  return  envelope.  A 
copy  "of  the  cover  letter  is  contained  in  Appendix  A.  The 
cover  letter  requested  that  the  surveys  be  returned  within 
seven  days  of  receipt;  however,  responses  were  accepted  as 
late  as  three  weeks  after  mailing. 

Analysis  of  Results .  The  survey  responses  were 
analyzed  using  the  SPSSx  statistical  package  on  the  AFIT 
Academic  Support  Computer.  The  following  is  a  brief 
overview  of  the  statistical  techniques  used.  The  analysis 
will  be  explained  in  greater  detail  in  Chapter  IV  -  Findings 
and  Analysis.  As  the  survey  responses  were  received,  they 
were  coded  into  an  SPSSx  data  file  by  category  of  response 
—  cost,  technical,  or  both.  This  was  done  to  ensure  that 
separate  statistical  analysis  could  be  accomplished  on  each 
category  of  data. 
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Several  statistical  techniques  were  used  to  analyze  the 
data.  The  techniques  chosen  were  based  on  the  level  of 
measurement  of  the  survey  responses  (9:1-22;  13:588-602). 
According  to  Emory,  there  are  four  widely  used 
classifications  of  measurement  scales:  nominal,  ordinal, 
interval,  and  ratio  (6:87).  The  level  of  measurement  is  a 
function  of  the  ordering  and  distance  properties  of  the 
data.  When  nominal  level  data  is  used,  the  responses  are 
partitioned  into  subsets  or  categories  which  are  "mutually 
exclusive  and  collectively  exhaustive"  (6:87).  Nominal 
level  data  includes  the  nominal  level  characteristics  plus 
an  indication  of  order  (greater  than  or  less  than).  All  of 
the  data  associated  with  the  survey  responses  was  nominal 
(lowest  level)  or  ordinal  level,  because  the  responses  were 
primarily  demographic  and  opinion  data. 

Two  techniques  were  selected  for  analysis  of  the  data: 
FREQUENCIES  and  CROSSTABS.  "FREQUENCIES  produces  a  table  of 
frequency  counts  and  percentages  for  the  values  of 
individual  variables"  (12:211).  The  FREQUENCIES  procedure 
quickly  tabulates  the  number  of  times  each  response  occurs 
and  displays  the  results  in  tabular  form.  The  tables  are 
used  to  determine  how  the  responses  are  distributed.  Once 
the  distribution  characteristics  of  the  responses  are  known, 
analysis  of  the  relationships  between  the  data  fields  can  be 
done.  One  procedure  for  analyzing  these  relationships  is 
called  CROSSTABS.  CROSSTABS  produces  a  contingency  table 
which  illustrates  a  row  by  column  representation  of  the 
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frequency  of  occurrence  between  different  categories  of  two 

data  fields.  "In  a  crosstabulation,  the  frequency 
distribution  of  one  variable  is  subdivided  according  to  the 
values  of  another  variable"  (16:87).  CROSSTABS  was  used  to 
examine  the  relationship  between  certain  characteristics  of 
the  respondents  and  their  opinions  or  preferences  concerning 
different  aspects  of  source  selection  training.  For 
example,  CROSSTABS  would  produce  a  contingency  table  of 
source  selection  experience  by  recommended  length  of  source 
selection  training.  The  table  would  have  the  experience 
level  (number  of  source  selections  the  respondent 
participated  in)  above  the  columns  and  the  recommended 
length  of  source  selection  training  courses  (in  days)  down 
the  side  of  the  table.  The  intersection  of  a  row  and  a 
column  would  contain  a  percentage  of  the  respondents  with  a 
certain  experience  level  who  chose  a  certain  length  of 
training  course. 

Additional  statistics,  such  as  measures  of  central 
tendencies,  were  not  generated  because  they  are  not 


applicable  to  nominal  or  ordinal  level  data.  A  oneway 
analysis  of  variance  using  the  SPSSx  technique  ONEWAY  was 
attempted  on  various  data  fields  that  could  be  interpreted 
as  interval  level  data  (dichotomies),  however,  no 
significant  results  were  achieved  (11:5). 
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IV.  Findings  and  Analysis 


Introduction 

This  chapter  presents  the  findings  of  the  three  phases 
of  research  described  in  Chapter  III  and  provides  an 
analysis  of  those  findings  as  they  relate  to  the  research 
objectives  presented  in  Chapter  I.  Phase  I  of  this  research 
identified  the  respondents  for  the  mail  survey.  Phase  II 
involved  a  literature  search  to  determine  the  available 
source  selection  training  programs,  and  Phase  III  was  the 
actual  mail  survey.  The  response  rate,  respondent 
background  information,  and  the  findings  required  to  answer 
the  research  objectives  are  presented  in  this  chapter.  Each 
research  objective's  analysis  contains  the  survey  questions 
associated  with  the  objective,  and  the  statistical 
techniques  applied  to  the  question. 

Phase  1^  -  Identify  Survey  Respondents 

Through  personal  interviews,  telephone  interviews,  and 
correspondence,  205  survey  respondents  were  identified 
throughout  the  AFSC  Product  Divisions.  These  respondents 
had  participated  in  the  Gray  and  Hugo  survey  or  had  been 
identified  by  the  various  Product  Divisions  as  having  had 
some  source  selection  experience. 

Phase  II  -  Availability  of  Training 

A  literature  search  was  conducted  to  determine  the 
availability  of  training  programs  covering  source  selection. 
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Overall,  very  little  training  exists  specifically  cov  i  i  r: 
the  source  selection  process.  Several  AF1T  Professional 
Continuing  Education  ( PCE )  courses  mention  the  source 
selection  process  as  a  means  of  selecting  a  qualified 
contractor,  but  do  not  cover  the  evaluation  process  in 
detail.  No  courses  were  identified  that  specifically 
provided  training  in  the  technical  evaluation  of  contract 
proposals . 


Phase  III  -  Mai 1  Survey  Questionnaire 

Once  the  respondents  were  identified,  and  the  decision 
made  to  conduct  a  mail  survey,  the  questionnaire  was 
administered  following  these  four  steps: 

1.  the  survey  questionnaire  was  constructed; 

2.  Air  Force  approval  to  administer  the  questionnaire 
was  obtained; 

3.  the  survey  was  mailed  to  the  205  respondents;  and 

4.  the  responses  were  analyzed. 

Step  One  -  Survey  Construction .  The  survey  was 
constructed  and  tested  during  January  through  March  1986. 
Several  modifications  were  made  during  this  period  based  on 
comments  received  from  the  thesis  advisor,  other  AFIT 
faculty,  and  other  graduate  students  familiar  with  the 
source  selection  process. 

Step  Two  -  Survey  Approval .  Once  the  survey  was 
completed,  the  survey  approval  request  package  was  sent  to 
the  Air  Force  Survey  Control  Office.  The  approval  process 
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rook  approximately  six  weeks,  due  in  part  to  the  Survey 
Control  Office  request  to  combine  the  cost  and  technical 
surveys.  Survey  approval  was  granted  on  6  June  1986  and 
survey  control  number  86-79  assigned.  The  survey  control 
number  appeared  on  each  survey  cover  letter. 

Step  Three  -  Survey  Administration .  A  survey  package 
consisting  of  the  survey,  the  cover  letter,  and  a  return 
envelope  was  mailed  to  each  of  the  respondents  on  26  June 
1986.  The  respondents  were  asked  to  complete  the  survey 
within  one  week  of  receipt;  however,  a  cut-off  date  of  23 
July  86  was  established  for  return  of  the  survey  responses 
to  allow  for  mailing  and  other  delays. 

Step  Four  -  Analysis  of  Results .  Since  the  survey 
questionnaire  requires  responses  in  four  distinct  sections, 
the  analysis  of  the  responses  was  also  divided  into  four 
sections : 

1.  response  rate  and  background; 

2.  general  source  selection; 

3.  analysis  of  cost  responses;  and 

4.  analysis  of  technical  responses. 

The  first  section  presents  the  response  rate  and  the 
demographic/background  information  of  the  respondents.  The 
second  section  presents  the  respondents  '  opinions  and 
preferences  regarding  the  overall  source  selection  process 
and  training  requirements.  The  third  section  on  cost  is  not 
covered  in  this  thesis  but  is  presented  in  a  thesis  entitled 
An  Assessment  of  Source  Selection  Cost  Panel  Members ' 
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P’*-  :  '-uv- dr. css  by  Captain  Steven  Babcock  (3).  The  final 
section  presents  the  respondents'  opinions  and  preferences 
regarding  the  technical  aspects  of  source  selection  and  the 
corresponding  training  requirements. 

The  descriptive  statistics  in  this  chapter  are 
presented  in  tabular  form  and  several  responses  may  be 
combined  and  included  in  the  same  table. 


Response  Rate  and  Background 


This  section  contains  a  summary  of  the  responses  to 
Part  1  (Background)  questions  from  the  survey. 

Response  Rate .  The  survey  was  sent  to  205  source 
selection  personnel,  and  124  returned  a  completed  survey  for 
an  overall  response  rate  of  60.5  percent.  Sixty  of  the  124 
respondents  had  been  members  of  a  technical  panel.  Table 
4.1  and  4.2  summarize  the  response  rate  by  Product  Division 
and  by  Military  Rank/Civilian  Grade  respectively. 

TABLE  4.1 

Product  Division  Response  Rate 


PRODUCT 

DIVISION 

SURVEYS 

MAILED 

COMPLETED 

SURVEYS 

RESPONSE 
RATE  (%) 

ASD 

94 

61 

64.9 

ESD 

44 

27 

61.4 

SD 

30 

14 

46.7 

AD 

22 

11 

50.0 

BMO 

11 

8 

72.7 

OTHER 

4 

3 

75.0 

TOTAL 

205 

124 

60.5 
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TABLE  4.2 


Military  Rank  or  Civilian  Grade  of  Survey  Respondents 


RANK/GRADE 

NUMBER 

PERCENT  OF  TOTAL 

COL 

5 

4.0 

LTC 

12 

9.7 

MAJ 

8 

6.5 

CAPT 

13 

10.5 

1LT 

6 

4.8 

2LT 

4 

3.2 

SES 

2 

1.6 

GM-15 

12 

9.7 

GM-14 

14 

11.3 

GM-13 

12 

9.7 

GS-14 

1 

.8 

GS-13 

12 

9.7 

GS-12 

16 

12.9 

GS-11 

1 

.8 

GS-9 

1 

.8 

GS-5 

2 

1.6 

No  grade 

provided 

3 

2.4 

TOTAL 

124 

100% 

Background  Information .  Tables  4.3  and  4.4 
summarize  how  familiar  the  respondents  were  with  the  various 
phases  of  the  acquisition  process  and  which  type  of 
acquisition  they  were  most  familiar  with.  Note  that  more 
than  one  response  could  have  been  chosen. 
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TABLE  4.5 

Current  Functional  Area  of  Survey  Rost  t  ndonts 


FUNCTIONAL  AREA 

NUMBER 

PERCENT  OF  TOTAL 

Program/Project  Mgmt 

33 

26 . 6 

Contracting  or 

Manufacturing  Mgmt 

34 

27 . 4 

Engineering 

12 

9 . 7 

Configuration  Mgmt 

2 

1.6 

Program  Control  or 
Business  Mgmt 

6 

4.8 

Logistics 

6 

4.8 

Test  and  Evaluation 

6 

4 . 8 

Cost  Analysis 

13 

10.8 

Other 

12 

9.7 

TOTAL 

124 

100% 

TABLE  4.6 

Education  Level  of  Survey  Respondents 


DEGREE 

NUMBER 

PERCENT  OF  TOTAL 

High  School 

7 

5.6 

Associate 

3 

2.4 

Bachelors 

38 

30.6 

Masters 

64 

51.6 

Doctorate 

12 

9.7 

TOTAL 

124 

100% 

Tables  4.7  through  4.9  show  the  experience  level  of  the 
survey  respondents  by  illustrating  the  number  of  years  of 
experience  as  well  as  the  actual  number  of  source  selections 
and  technical  panels  that  the  respondents  have  participated 


in . 
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TABLE  4.7 


Years  of  Source  Selection 

Experience 

YEARS  NUMBER 

PERCENT  OF  TOTAL 

0-2 

39 

31.5 

2-5 

31 

25.0 

5-10 

21 

16.9 

10-15 

17 

13.7 

15-20 

6 

4.8 

More  than  20 

10 

8.1 

TOTAL 

124 

100% 

TABLE  4.8 

Number  of  Source  Selections  Survey  Respondents  Have 

Participated  In 


NUMBER  OF 

SOURCE  SELECTIONS 

NUMBER 

PERCENT  OF  TOTAL 

None 

2 

1.6 

1  to  4 

64 

51.6 

5  to  7 

20 

16.1 

8  or  greater 

38 

30.6 

TOTAL 


124 


100% 


TABLE  4.9 


n 


\ 

j 

1 


( 

i 

i 

I 

* 

% 

4 


I 

f 

t 
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Number  of  Technical  Panels  Survey  Respondents  Have  Been 

A  Member  Of 


NUMBER  OF 

TECHNICAL  PANELS 

NUMBER 

PERCENT  OF  TOTAL 

None 

64 

51 . 6 

1  to  4 

39 

31.4 

5  to  7 

11 

8.8 

8  or  greater 

10 

8.1 

TOTAL 

124 

100% 

Research  Objective  1 

Determine  if  any  general  or  selection-specific 
education  was  received  by  source  selection 
participants  prior  to  a  source  selection. 

•Survey  Questions  and  Statistical  Techniques .  Survey 
questions  12  through  15  (see  Appendix  B  for  copy  of  survey) 
provided  the  responses  necessary  to  answer  the  research 
objective.  The  SPSSx  routine  FREQUENCIES  generated  the 
percentage  of  occurrence  of  each  response  for  the  applicable 
question. 

Findings  and  Analysis .  Survey  questions  14  and  15 
assessed  whether  the  respondents  had  received  any  source 
selection  training  prior  to  their  first  or  subsequent  source 
selection  participation.  Table  4.10  indicates  that  28  of 
the  124  respondents  (22.6%)  had  received  some  source 
selection  training  prior  to  their  first  source  selection. 
Table  4.11  indicates  that  an  additional  25  respondents 
received  training  prior  to  subsequent  source  selection 
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TABLE  4.10 


Did  Survey  Respondents  Receive  Source  Selection  Training 
Prior  to  Their  First  Source  Selection? 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

Yes 

28 

22.6 

No 

96 

77.4 

TOTAL 

124 

100% 

The  majority  of  the  respondents  who  answered  yes  to 
having  received  training  prior  to  their  first  source 
selection  (Table  4.10),  received  their  training  from  the 
Product  Division  source  selection  office  in  the  form  of  a 
briefing  lasting  less  than  one  day. 

TABLE  4.11 

Did  Source  Selection  Participants  Receive  Any  Source 
Selection  Training  Prior  To  Subsequent  Source  Selections? 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

Yes 

25 

20.2 

No 

73 

58.9 

No  Response 

26 

21.0 

TOTAL 

124 

100% 

The  majority  of  any  subsequent  training  received  by  the 
respondents  was  also  in  the  form  of  a  briefing  by  the  source 
selection  office  lasting  less  than  one  day. 

Technical  Panel  Members .  Sixty  of  the  124 


respondents  (48%)  indicated  that  they  had  served  on  a 
technical  panel.  Fifteen  of  the  60  technical  panel  members 
(25%)  received  some  formal  source  selection  training  prior 
to  serving  on  their  first  technical  panel.  Sixteen 
technical  panel  members  received  training  prior  to  a 
subsequent  source  selection.  These  findings  are  consistent 
with  the  overall  sample  population.  In  general,  the 
findings  of  technical  panel  members  were  consistent  with  the 
findings  of  the  overall  sample  population. 

Survey  question  12  determined  the  source  of  the 
respondents'  source  selection  knowledge.  A  majority  of  the 
respondents  indicated  that  they  received  their  knowledge 
from  previous  source  selections  or  through  on-the-job 
training  as  indicated  in  Table  4.12. 

TABLE  4.12 

From  What  Source  Did  the  Survey  Participants  Receive  Their 

Source  Selection  Knowledge? 


SOURCE  OF  KNOWLEDGE 

NUMBER 

PERCENT  OF  TOTAL  * 

Formal  Training  Course 

7 

5.6 

On-the-Job  Training 

73 

58.9  • 

Self-Study  Course 

7 

5.6 

No  Experience 

3 

2.4 

Previous  Source  Selections 

70 

56.5 

Other  Sources 

3 

2.4 

*  NOTE:  Multiple  Answers 

were  possible. 

Table  4.13  presents  the  formal  Air  Force  training 
courses  that  the  respondents  have  taken  that  contain 
material  on  the  source  selection  process.  The  majority  of 


t respondents  have  taken  at  least  one  course  that  contains 
material  on  the  source  selection  process,  however,  none  of 
the  courses  listed  cover  the  source  selection  process  or 
procedures  in  any  great  detail. 

TABLE  4.13 

Formal  Air  Force  Training  Courses  Containing  Source 
Selection  Material  that  Survey  Respondents  Have  Attended 


TRAINING  COURSE 

NUMBER 

PERCENT  OF  TOTAL  * 

None 

60 

48.4 

Systems  200/400 

15 

12.1 

Contract  Administration 

(PPM  152) 

13 

10.5 

Logistics  Management 

(Log  224) 

2 

1.6 

Government  Contract  Law 

(PPM  302) 

20 

16.1 

Tech,  Cost,  Price  Analysis 

(QMT  345) 

14 

11.3 

Other  Courses 

25 

20.2 

*  NOTE:  Multiple  answers  were  possible. 


Research  Objective  2 

Determine  what  courses  currently  exist  that 
specifically  cover  or  include  material  on  the 
source  selection  process. 

A  literature  search  and  a  review  of  Air  Force  Institute 
of  Technology  (AFIT)  courses  was  conducted  to  determine  the 
availability  of  courses  covering  the  source  selection 
process . 

Findings  and  Analysis.  Overall,  very  little  training 
exists  specifically  covering  the  details  of  the  source 
selection  process.  The  following  AFIT  Professional 
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Continuing  Education  ( PCE )  courses  cover  the  source 
selection  process  as  a  means  of  selecting  a  qualified 
contractor,  but  do  not  cover  the  evaluation  process  in 
detail . 

1.  Systems  200/400 

2.  Contract  Administration  (PPM  152) 

3.  Logistics  Management  (LOG  224) 

4.  Government  Contract  Law  (PPM  302) 

5.  Technical,  Cost,  Price  Analysis  (QMT  345) 

6.  Advanced  Topics  in  Acquisition  Management  (AMGT  655) 
(APIT  Graduate  School  elective) 


Business  Management  Research  Associates  Inc.  (BMRA) 
under  contract  to  the  Air  Force  Business  Management  Research 
Center  ( AFBMRC )  identified  the  following  courses  throughout 
the  Department  of  Defense  that  include  material  on  source 
selection  (4:1-13). 

Title  Offered  by 

Management  of  Defense  ALMC,  Ft.  Lee 

Acquisition  Contracts 

Defense  Contracts  DMET  (BMRA) 

Management  for  Technical 
Personnel 

Source  Evaluation  Procedures  GSA 

Contracting  Specialist  ATC,  Lowry  AFB 


vv 


Most  of  the  respondents  indicated  that  their  training 
consisted  of  a  briefing  given  by  the  Source  Selection  Office 
lasting  less  than  one  day. 

Research  Object ive  3 

Determine  if  a  source  selection  training  course 
would  be  beneficial  to  technical  panel 
participants  or  source  selection  participants  in 
general . 

Survey  Questions  and  Statistical  Techniques .  Survey 
questions  16,  35,  36,  and  37  provided  the  necessary 
responses  to  accomplish  this  research  objective.  The  SPSSx 
routines  FREQUENCIES  and  CROSSTABS  were  used  to  analyze  the 
responses.  CROSSTABS  was  used  to  determine  if  the 
experience  level  of  the  respondents  had  any  influence  on  how 
beneficial  they  felt  a  source  selection  training  course 
might  be. 

Findings  and  Analysis .  Table  4.14  indicates  that  56% 
of  the  respondents  would  attend  a  source  selection  training 
course  if  one  was  available. 

TABLE  4.14 

Would  Survey  Respondents  Attend  a  Source  Selection 
Training  Course  if  One  Was  Available? 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

Yes 

70 

56.5 

No 

54 

43.5 

TOTAL 

124 

100% 
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Table  4.15  is  a  contingency  table  showing  the  number  of 
respondents  willing  to  attend  a  source  selection  training 
course  against  the  background  variable  experience.  As 
expected,  the  more  experience  a  respondent  had,  the  less 
willing  he  was  to  attend  training. 

TABLE  4.15 


Contingency  Table  -  Source  Selection  Experience  by 
Willingness  to  Attend  Source  Selection  Training 


COUNT  I 
ROW  PCT  I 
COL  PCT  I 
TOT  PCT  I 

- 1 

I 

0-2  YRS  I 

I 
I 

- ! 

I 

2-5  YRS  I 

I 
I 

- 1 

I 

5-10  YRS  I 

I 
I 

- 1 

I 

10-15  YRS  I 
I 
I 

- j 

I 

15-20  YRS  I 
I 
I 

- 1 

I 

>  20  YRS  I 
I 
I 

- X 

COLUMN 

TOTAL 


YES  I  NO 

- 1 - 

31  I  8 

79.5  I  20.5 

44.3  I  14.8 

25.0  I  6.5 

- 1 - 

18  I  13 

58.1  I  41.9 

25.7  I  24.1 

14.5  I  10.5 

- 1 - 

12  1  9 

57.1  I  42.9 

17.1  I  16.7 

9.7  I  7.3 

- 1 - 

6  1  11 

35.3  I  64.7 

8.6  I  20.4 

4.8  I  8.9 

- 1 - 

1  I  5 

16.7  I  83.3 

1.4  I  9.3 

.8  I  4.0 

- 1 - 

2  1  8 

20.0  I  60.0 

2.9  I  14.8 

1.6  I  6.5 

- 1 - 

70  54 

56.5  43.5 


I 

I- 

I 

I 

I 

I 

I- 

I 

I 

I 

I 

I- 

I 

I 

I 

I 

I- 

I 

I 

I 

I 

I- 

I 

I 

I 

I 

I- 

I 

I 

I 

I 

IT- 


ROW 

TOT 


39 

31.5 


31 

25.0 


21 

16.9 


17 

13.7 


6 

8 


10 

8.1 


124 

100.0 


Table  4.16  shows  how  beneficial  a  form.al  course 


covering  the  overall  source  selection  process  and  specific 
procedures  would  be  to  technical  panel  members.  The 
majority  of  the  respondents  felt  that  such  a  course  would  be 
beneficial  to  them. 

TABLE  4.16 


How  Beneficial  the  Respondents  Felt  a  Formal  Course  Would  Be 


In  Providing  Information 

and 

on  the  Source  Selection  Process 

Procedures 

RESPONSE 

NUMBER 

PERCENT  OF  TOTAL  . 

1 —Not  beneficial 

0 

0.0 

2  * 

3 

5.0 

3-Somewhat  beneficial 

17 

28.3 

4  * 

24 

40.0 

5-Very  beneficial 

15 

25.0 

No  response 

1 

1.7 

*  NOTE:  A  five  point  Likert  scale  was 
for  responses  1,  3,  and  5. 

used  with  headings 

TOTAL 

60 

100% 

Table  4.17  shows  how  beneficial  technical  panel  members 
felt  a  contractor  presentation  would  be  in  shortening  the 
evaluation  process  or  resolving  proposal  deficiencies’.  Such 
a  presentation  would  not  replace  a  formal  course,  but  could 
be  used  to  supplement  a  course  by  providing  the  necessary 
technical  information  during  a  specific  source  selection. 
Again,  most  respondents  felt  that  a  contractor  presentation 
would  be  beneficial. 
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TABLE  4.17 


How  Beneficial  Respondents  Felt  a  Contractor  Presentation 
Would  Be  During  the  Technical  Evaluation  Process 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

1-Not  beneficial 

8 

13.3 

2  * 

6 

10.0 

3-Somewhat  beneficial 

16 

26.7 

4  * 

14 

23.3 

5-Very  beneficial 

14 

23.3 

No  response 

2 

3.3 

*  NOTE:  A  five  point  Likert 

scale  was 

used  with  headings 

for  responses  1,  3,  and 

5. 

TOTAL 

60 

100% 

Table  4.18  presents 

how 

beneficial 

a  formal  course 

providing  specific  technical 

background 

would  be  to 

technical  panel  members. 

In 

this  case, 

most  respondents 

felt  that  such  a  course  would  not  be  of  much  benefit  to 
them.  Based  on  the  responses  and  comments  received,  most 
respondents  felt  that  they  would  not  be  on  the  technical 
panel  if  they  did  not  have  the  required  technical 
background.  Therefore,  they  felt  that  a  formal  course 
covering  technical  topics  would  be  of  little  value  to  them. 


TABLE  4.18 


How  Beneficial  the  Respondents  Felt  a  Formal  Course  Would  Be 
in  Providing  Panel  Members  the  Required  Technical  Background 


RESPONSE  NUMBER 

PERCENT  OF  TOTAL 

1-Not  beneficial 

13 

21.7 

2  * 

12 

20.0 

3-Somewhat  beneficial 

13 

21.7 

4  * 

11 

18.3 

5-Very  Beneficial 

8 

13.3 

No  response 

3 

5.0 

*  NOTE:  A  five  point  Likert 
for  responses  1,  3,  and  5. 

scale  was 

used  with  headings 

TOTAL 

60 

100% 

Research  Objective  4^ 

Determine  the  format  and  length  of  any  source 
selection  training  course  if  the  need  for  one  is 
identified. 

The  previous  research  objective  showed  that  the 
majority  of  the  respondents  felt  that  a  training  course 
covering  the  general  source  selection  process  and  procedures 
would  be  of  benefit  to  them.  This  objective  presents  the 
respondents  feelings  concerning  the  content  and  length  of  a 
source  selection  training  course  and  who  should  be 
responsible  for  conducting  the  training. 

Survey  Questions  and  Statistical  Techniques .  Survey 
questions  17  through  20,  38,  and  39  provided  the  necessary 
responses  to  accomplish  this  research  objective.  The  SPSSx 
routine  FREQUENCIES  was  used  to  analyze  the  responses. 

Findings  and  Analysis.  Technical  panel  members  were 
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asked  whether  or  not  the  source  selection  schedule  would 
'♦I  low  for  any  courses  to  be  conducted.  Table  4.19  shows 
that  the  respondents  were  almost  split  in  their  opinions, 
however,  a  slight  majority  felt  that  the  schedule  would 
allow  for  some  training  to  be  conducted. 

TABLE  4.19 

Would  the  Technical  Panel  or  Source  Selection  Schedule 
Allow  for  a  Training  Course  to  be  Conducted? 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

Yes 

33 

55.0 

No 

26 

43.3 

No  response 

1 

1.7 

TOTAL 

60 

100% 

All  of  the  the  survey  participants  were  asked  what  they 
felt  the  optimum  length  for  any  formal  course  should  be. 
Table  4.20  shows  the  majority  of  the  respondents  felt  that 
the  length  of  any  course  should  be  held  to  one  week  or  less. 
Nearly  40%  of  the  respondents  indicated  that  the  optimum 
length  for  a  source  selection  course  should  be  1-2  days. 

The  responses  of  the  technical  panel  members  were  consistent 
with  the  overall  sample  population.  Twenty-six  out  of  60 
respondents  felt  that  1-2  days  would  be  the  optimum  length 
for  a  source  selection  course. 
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TABLE  4.20 


Length  of  the  Source  Selection  Course 


LENGTH 

NUMBER 

PERCENT  OF  TOTAL 

Less  than  a  day 

9 

7.3 

1-2  days 

48 

38.7 

3-5  days 

33 

26.6 

1  week 

23 

18.5 

2  weeks 

7 

5.6 

Other 

4 

3.2 

TOTAL 

124 

100% 

Table  4.21  presents  what  topics  the  respondents  felt 
should  be  included  in  a  source  selection  training  course. 

A  majority  of  the  respondents  indicated  that  all  the  topics 
listed  should  be  included,  but  indicated  a  higher  preference 
for  including  general  source  selection  procedures  and 
instructions  on  evaluating  contractors'  proposals.  Again, 
the  technical  panel  members'  responses  were  consistent  with 
the  overall  sample  population. 

TABLE  4.21 

The  Content  of  the  Source  Selection  Training  Course 

Should  Include 


COURSE  CONTENT 

NUMBER 

PERCENT  OF  1 

Instructions  on  the  RFP 

71 

57.3 

Instructions  on  Forms 
General  Source 

61 

49.2 

-  Selection  Procedures 
Instructions  on  Evaluating 

98 

79.0 

Contractors  Proposals 

98 

79.0 

Other 

15 

12.1 

*  NOTE:  Multiple  answers  were  possible. 


Table  4.22  shows  tne  source  selection  course  format 

recommended  by  the  respondents.  Once  again,  the  respondents 
indicated  their  preference  for  a  course  covering  general 
source  selection  procedures. 

TABLE  4.22 

Format  for  the  Source  Selection  Training  Course 


FORMAT 

NUMBER 

PERCENT  OF  TOTAL 

Selection  specific 
(short  course  prior  to 
specific  SS  start) 

General  (short  course 

46 

37.1 

on  general  procedures) 
Video  or  sound-on- 

69 

55.6 

slide  presentation 

21 

16.9 

Other  format 

6 

4.8 

*  NOTE:  Multiple  answers 

were  possible. 

Table  4.23  indicates  the  respondents'  preference  for 
who  should  conduct  the  source  selection  training  course. 

The  majority  of  the  respondents  felt  that  the  course  should 
be  conducted  by  the  product  divisions  or  the  source 
selection  office  within  each  product  division. 

TABLE  4.23 

Source  Selection  Training  Course  Should  Be  Given  By 


ORGANIZATION 

NUMBER 

PERCENT  OF  TOTAL 

Product  Division 

44 

35.5 

Individual  SPO 

16 

12.9 

Source  Selection  Office 

35 

28.2 

Other 

29 

23.4 

TOTAL 

124 

100% 
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Research  Objective  5 


Determine  the  level  of  expertise  required  by 
regulation  for  membership  on  a  technical  panel. 

Findings  and  Analysis .  Air  Force  Regulation  70-15, 

Source  Selection  Policy  and  Procedures  is  the  guiding 

directive  governing  the  source  selection  process.  This 

regulation  states  the  general  level  of  expertise  required 

for  membership  on  any  panel  comprising  the  SSEB.  "The  SSEB 

should  be  formed  of  fully  qualified  government  personnel 

possessing  the  professional  skills  and  knowledge  required 

for  an  evaluation  and  assessment  of  offerors'  proposals" 

(5:4).  The  regulation  goes  on  to  state: 

Evaluators  are  expected  to  understand  the  requirement, 
the  solicitation,  the  evaluation  criteria,  and  the 
evaluation  standards.  They  must  also  be  well  versed  in 
’their  fields.  When  it  is  necessary  to  verify  certain 
aspects  of  proposals  outside  their  technical  skill, 
evaluators  are  encouraged  to  engage  in  discussions 
with  advisors,  or  other  SSEB  members  [5:10]. 

Most  technical  panel  members  surveyed  felt  they  had  the 

required  technical  background  to  perform  their  duties  (see 

Research  Objective  3).  This  is  consistent  with  the  general 

requirement  stated  in  the  regulation  that  the  evaluators  be 

well  versed  in  their  field.  The  survey  responses  indicate, 

however,  that  although  technical  panel  members  are 

technically  qualified,  additional  training  covering  source 

selection  procedures  (requirements,  criteria,  standards, 

etc.)  would  be  beneficial. 
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Determine  how  technical  panel  members  are  chosen. 

This  research  objective  assesses  how  technical  panel 
members  are  selected.  Technical  panel  members  were  asked 
how  they  were  selected  for  their  responsibilities  on  the 
panel,  and  how  they  select  members  for  technical  panels  if 
required  to  do  so. 

Survey  Questions  and  Statistical  Techniques .  Survey 
questions  29,  30,  and  31  provide  the  necessary  responses  to 
accomplish  this  research  objective.  The  SPSSx  routines 
FREQUENCIES  and  CROSSTABS  were  used  to  analyze  the 
responses.  CROSSTABS  was  used  to  determine  if  source 
selection  experience  had  any  influence  on  how  people  made 
selections  for  technical  panels.  For  example,  if  a  person 
had  10  years  of  source  selection  experience,  did  that  affec 
the  way  he  selected  technical  panel  members? 

Findings  and  Analysis .  Table  4.24  presents  the 
perceptions  of  the  technical  panel  members  on  hew  they  were 
selected  for  their  technical  panel  responsibilities.  The 
majority  of  the  respondents  indicated  that  their  membership 
on  the  technical  panel  was  due  to  a  combination  of  factors 
including  the  training  they  received,  their  experience 
level,  and  their  expertise  in  a  specific  area. 


TABLE  4.24 


How  Individuals  Were  Selected  for  a  Technical  Panel 


HOW  SELECTED  NUMBER  PERCENT  OF  TOTAL  * 


Past  experience  33  55.9 
Education  level  6  10.0 
Training  received  59  98.3 
Expertise  in  specific  area  37  61.7 
Random  selection  3  5.0 
Only  one  available  7  11.9 
Part  of  my  job  42  70.0 
Other  2  3.3 


*  NOTE:  Multiple  answers  were  possible. 


Table  4.25  shows  how  the  survey  respondents  select 
individuals  for  technical  panels.  Although  technical 
competence  was  indicated  most  often  by  the  respondents,  one- 
third  said  that  they  rely  on  the  recommendations  of  others. 
Thus,  if  a  technical  panel  member  performs  well  during  a 
source  selection,  he  or  she  may  have  a  greater  chance  of 
being  selected  for  another. 

TABLE  4.25 

How  the  Survey  Respondents  Select  Individuals  for 
Membership  on  a  Technical  Panel 


HOW  SELECTED 

NUMBER 

PERCENT  OF  TOTAL  * 

Technical  background 

38 

63.3 

Education  level 

5 

8.3 

Training  received 

7 

11.7 

Recommendation  by  others 

20 

33.3 

Randomly 

1 

1.7 

Other 

12 

20.0 

*  NOTE:  Multiple  answers 

were  possible. 

Table  4.26  indicates  what  the  respondents  felt  would  be 


the  ideal  way  technical  panel  members  should  be  selected. 
Again,  technical  background  was  listed  as  being  the  most 
important  criterion,  and  reliance  on  recommendations  by 
others  was  still  considered  highly  credible. 

TABLE  4.26 

How  the  Respondents  Felt  Technical  Panel  Members  Should 

Be  Selected 


SELECT  BY 

NUMBER 

PERCENT  OF  TOTAL  * 

Technical  background 

55 

91.7 

Education  level 

6 

10.0 

Training  received 

19 

31.7 

Recommendation  by  others 

25 

41.7 

Randomly 

0 

0.0 

Other 

5 

8.3 

*  NOTE:  Multiple  answers 

were  possible. 

Research  Objective  7 

Determine  how  well  prepared  members  cf  previous 
technical  panels  were  for  their  responsibilities 
on  the  panel . 

This  research  objective  will  assess  how  well  prepared 
previous  technical  panel  members  were,  and  determine  what 
they  felt  were  the  biggest  problems  they  encountered. 

Survey  Questions  and  Statistical  Techniques .  Survey 
questions  21,  22,  32,  33,  and  34  provided  the  necessary 
responses  to  accomplish  this  research  objective.  The  SPSSx 
routines  FREQUENCIES  and  CROSSTABS  were  used  to  analyze  the 
responses . 

Findings  and  Analysis .  The  survey  participants  were 


asked  that  if  they  received  formal  source  selection 
training,  did  that  training  adequately  prepare  them  for 
their  source  selection  responsibilities?  This  question 
was  asked  in  an  effort  to  assess  the  effectiveness  and 
utility  of  existing  source  selection  courses,  as  well  as  to 
determine  the  preparedness  of  the  participants.  Table  4.27 
shows  that  the  respondents  were  mixed  in  their  opinions  as 
tc  whether  the  formal  training  adequately  prepared  them  for 
their  responsibilities.  The  wide  variety  of  responses  makes 
suspect  the  effectiveness  of  existing  source  selection 
courses,  and  points  to  the  need  for  a  common  comprehensive 
course  throughout  AFSC  that  combines  the  strengths  of 
existing  courses. 

TABLE  4.27 

Did  Formal  Training  Adequately  Prepare  Respondents  for 

Source  Selection? 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

Stongly  disagree 

6 

4.8 

Disagree 

Neither  agree  nor 

8 

6.5 

disagree 

9 

7.3 

Agree 

12 

9.7 

Strongly  agree  2 

No  response  (or  no  formal 

1.6 

training  received) 

87 

70.2 

TOTAL 

124 

100% 

Table  4.28  shows  how  well  prepared  the  technical  panel 
members  felt  they  were  for  their  responsibilities.  Table 
4.29  shows  how  well  prepared  the  technical  panel  members 
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felt  other  members  of  the  panel  were  for  their 
responsibilities.  It  is  interesting  to  note  that  in  most 
cases  the  respondents  felt  that  they  were  better  prepared 
than  the  other  members  of  the  technical  panel.  In  general, 
however,  the  majority  of  the  technical  panel  members  felt 
prepared  for  their  responsibilities,  and  felt  that  their 
fellow  panel  members  were  prepared  as  well. 


TABLE  4.28 

How  Well  Prepared  Were  the  Respondents  for  Their 
Responsibilities  on  the  Technical  Panel? 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

Unprepared 

1 

1.7 

Somewhat  unprepared 
Neither  prepared 

13 

21.7 

nor  unprepared 

6 

10.0 

Somewhat  prepared 

23 

38.3 

Well  prepared 

16 

26.7 

No  response 

1 

1.7 

TOTAL 


60 


100% 


TABLE  4.29 


How  Prepared  the  Respondents  Felt  Other  Members  of  the 
Technical  Panel  Were  For  Their  Responsibilities 


RESPONSE 

NUMBER 

PERCENT  OF  TOTAL 

Unprepared 

3 

5.0 

Somewhat  prepared 

15 

25.0 

Neither  prepared 

nor  unprepared 

12 

20.0 

Somewhat  prepared 

21 

35.0 

Well  prepared 

8 

13.3 

No  response 

1 

1  .7 

TOTAL 

60 

100% 

Table  4.30  and  4.31  present  the  respondents'  opinions 
on  what  they  consider  tc  be  the  major  problems  encountered 
in  the  source  selection  process.  Table  4.30  presents  the 
opinions  of  the  entire  sample  population,  while  table  4.31 
presents  the  opinions  of  the  technical  panel  members  only. 
Nearly  half  of  the  overall  sample  population  listed 
inexperience  of  the  source  selection  personnel  as  being  the 
biggest  problem  they  encountered  as  a  source  selection 
participant. 
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TABLE  4.30 


The  Biggest  Problem  Encountered  as  a  Source  Selection  Member 


PROBLEM 

NUMBER 

PERCENT  OF  TOTAL  * 

Not  aware  of  source 

selection  procedures 

24 

19.4 

Interpreting  and 

filling  out  forms 

11 

8.9 

Quality  of  contractor 

information 

32 

25.8 

Time  restrictions 

28 

22.6 

Inexperience  of  source 

selection  personnel 

60 

48.4 

Not  familiar  with  RFP 

15 

12.1 

Other  problems 

17 

13.7 

*  NOTE:  Multiple  answers 

were  possible. 

TABLE  4.31 

What  the  Respondents  Felt  Were  the  Major  Problem  Areas 
in  the  Evaluation  of  Contractors'  Technical  Proposals 


PROBLEM  AREA 

NUMBER 

PERCENT  OF  TOTAL  * 

Development  of  standards 

37 

61.7 

Interpretation  of 

contractor  proposals 

21 

35.0 

Competence  of  participants 

17 

28.3 

Unfamiliarity  with 
technical  panel  procedures 

13 

21.7 

Administrative  problems 
(filling  out  forms) 

13 

21.7 

No  problems  exist 

4 

6.7 

*  NOTE:  Multiple  answers 

were  possible. 

Table  4.32  presents  the  results  of  the  SPSSx  routine 
CROSSTABS.  The  variables  examined  are  source  selection 
experience  versus  preparedness  for  a  technical  panel.  As 
expected,  the  more  experience  a  respondent  had,  the  better 


prepared  he  felt  he  was  for  his  technical  panel 
responsibilities . 

TABLE  4.32 

Contingency  Table  -  Experience  by  Technical  Panel 

Preparedness 
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*  1  -  unprepared 

2  -  somewhat  unprepared 

3  -  neither  prepared  nor  unprepared 

4  -  somewhat  prepared 

5  -  well  prepared 


V.  Cone  1  us  ions  and  Recommend at  ions 


Chapter  Overview 

This  chapter  presents  the  conclusions  that  can  be  drawn 
from  this  research  effort  on  the  source  selection  process. 
The  survey's  large  sample  size  and  high  response  rate 
provide  confidence  that  the  results  achieved  are  indicative 
of  the  entire  Air  Force  source  selection  population.  The 
numerous  comments  received  (see  Appendix  C)  demonstrate  that 
the  survey  participants  are  genuinely  concerned  about 
improving  the  source  selection  process.  The  fact  that  the 
respondents  took  the  time  to  complete  the  survey  and  provide 
their  comments  illustrates  the  importance  of  the  source 
selection  process  to  Air  Force  Systems  Command  and  the  Air 
Force  in  general. 

The  conclusions  and  recommendations  are  presented  in 
three  general  areas  --  preparedness  of  source  selection 
participants,  particularly  technical  panel  members;  benefit 
of  a  source  selection  training  course;  and  format,  content, 
and  length  of  a  source  selection  course. 

Preparedness  of  Source  Selection  Participants 

Conclusions .  The  following  is  a  list  of  the 
conclusions  drawn  from  the  survey  responses  concerning  the 
preparedness  of  source  selection  participants: 

1.  Source  selection  participants  as  a  whole  are 
not  receiving  adequate  training  on  general  source  selection 
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procedures  prior  to  their  participation  in  the  source 
selection  process.  Only  23  percent  of  the  source  selection 
participants  responding  to  the  survey  had  received  training 
prior  to  their  first  source  selection. 

2.  Technical  panel  members  feel  technically 
competent  and  adequately  prepared  to  evaluate  contract 
proposals;  however,  they  require  additional  training  on  the 
specific  source  selection  process  and  procedures. 

3.  In  general,  the  majority  of  the  survey 
respondents  felt  that  the  biggest  problem  encountered  as  a 
source  selection  participant  was  the  inexperience  of  other 
source  selection  personnel. 

4.  The  majority  of  the  survey  participants  gained 
their  source  selection  experience  from  on-the-job  training 
or  through  participation  in  previous  source  selections. 

Thus,  personnel  may  be  assigned  to  a  specific  source 


selection  for  the  purpose  of  gaining  experience  for  future 
source  selections,  or  just  to  learn  more  about  the  program. 

Discussion .  The  lack  of  source  selection  training  has 
a  definite  impact  on  the  source  selection  process.  The 
unfamiliarity  with  the  general  source  selection  process  and 
specific  procedures  has  diverted  attention  away  from  the 
important  task  of  effectively  evaluating  contract  proposals 
and  caused  unnecessary  delays  while  the  participants  are 

"  .  a 

informed  of  proper  procedures.  The  comments  received 
indicate  that  the  evaluation  process  is  frequently  delayed 
or  interrupted  due  to  inexperienced  or  unqualified 


personnel.  Training  beneficial  to  source  selection 
participants  would  include:  explanation  of  the  RFP, 
instruction  on  filling  out  the  source  selection  forms, 
explanation  of  the  evaluation  criteria  and  standards,  and 
instruction  on  how  to  evaluate  contractors'  proposals. 

The  survey  results  indicate  that  the  technical  panel 
members  feel  they  are  technically  qualified  to  evaluate  a 
contractor's  proposal.  The  majority  of  the  respondents 
commented  that  they  would  not  have  been  selected  for  the 
technical  panel  had  they  not  possessed  the  necessary 
expertise.  These  results  indicate  that  the  technical  panel 
members  are  effectively  fulfilling  their  technical 
responsibilities  during  the  source  selection  process.  The 
technical  panel  members  indicated,  however,  that  a  course 
covering  general  source  selection  procedures  would  be 
beneficial  to  them.  Their  responses  were  consistent  with 
the  overall  sample  population  concerning  the  benefit  of  a 
source  selection  training  course. 

Recommendations .  The  following  is  the  major 
recommendation  based  on  the  results  of  this  study  on  the 
preparedness  of  source  selection  participants. 

1.  A  comprehensive  source  selection  training 
course  should  be  developed.  The  course  should  explain  the 
source  selection  process,  structure,  and  procedures,  and 
include  topics  ranging  from  the  basic  Air  Force  source 
selection  philosophy  to  how  to  fill  out  the  basic  source 
selection  forms.  The  course  should  emphasize  tailoring  the 


5-3 


source  selection  process  to  a  particular  program.  The 
course  should  be  made  available  throughout  AFSC  and  taught 
in  conjunction  with  any  product  division  course  currently 
available.  The  recommended  specifics  of  the  course  are 
presented  later  in  this  chapter. 

Benefits  of  a  Source  Selection  Training  Course 

Conclusions .  The  following  is  a  list  of  conclusions 
drawn  from  the  research  data  concerning  the  benefits  of  a 
source  selection  training  course. 

1.  By  far,  the  majority  of  the  survey 
participants  indicated  that  a  source  selection  training 
course  would  be  beneficial  to  them. 

2.  Technical  panel  members  who  felt  prepared  for 
their  duties  indicated  that  a  source  selection  training 
course  would  be  beneficial  to  them,  and  provide  them  with  a 
better  overall  understanding  of  the  source  selection 
process . 

3.  A  source  selection  training  course  has  the 
potential  to  reduce  the  length  of  the  overall  source 
selection  process  by  better  preparing  the  participants  for 
their  responsibilities. 

Discussion.  Nearly  all  of  the  respondents  to  the 
survey  indicated  that  a  source  selection  training  course 
would  be  beneficial  in  some  manner.  Those  who  said  they 
would  not  attend  a  training  course  if  one  was  offered,  had 
extensive  source  selection  experience.  Without  exception, 


these  experienced  people  said  that  they  would  have  attended 
a  source  selection  training  course  earlier  in  their  careers. 
In  addition,  these  experienced  people  indicated  that  they 
would  encourage  any  inexperienced  personnel  to  attend  a 
training  course. 

The  source  selection  process  is  long  and  complicated. 

By  better  preparing  the  participants,  the  possibility  exists 
to  not  only  enhance,  but  shorten  the  process.  Better 
prepared  personnel  in  addition  to  improved,  tailored 
procedures  have  the  potential  to  significantly  improve  the 
source  selection  process  throughout  AFSC. 

Recommendations .  A  source  selection  training  course 
should  be  made  mandatory  for  all  first  time  source  selection 
participants.  Other  personnel  with  some  source  selection 
experience  should  be  encouraged  to  attend,  but  priority 
should  be  given  to  first  time  participants.  Ideally,  the 
course  should  be  conducted  as  close  to  the  start  of  the 
source  selection  as  possible  to  maximize  student  retention. 

Format ,  Content ,  and  Length  of  a  Training  Course 

Conclusions .  The  following  is  a  list  of  conclusions 
concerning  the  development  and  presentation  of  a  source 
selection  training  course. 

1.  The  length  of  a  source  selection  training 
course  should  be  one  week  or  less  so  as  not  to  interfere 
with  the  source  selection  process. 

2.  The  format  for  the  course  should  be  a  general 
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short  course  similar  to  the  AFIT  continuing  education 


courses . 


3.  The  course  should  include  but  not  be  limited 
to  the  following:  Air  Force  source  selection  philosophy, 
instructions  on  the  SSP  and  RFP ,  general  source  selection 
procedures,  color  coding,  and  instructions  on  criteria, 
standards,  and  evaluating  contractors'  proposals. 

4.  The  course  should  be  conducted  by  a  central 
organization  to  provide  uniformity  and  continuity.  The 
product  division  or  the  source  selection  office  within  the 
product  division  could  sponsor  the  course  in  conjunction 
with  previously  established  courses  or  briefings. 

Discussion .  The  survey  participants  made  very  specific 
recommendations  regarding  the  development  and  presentation 
of  a  source  selection  training  course.  These 
recommendations  varied,  however,  due  to  the  experience  level 
of  the  respondents.  Business  Management  Research 
Associates  Inc.  ( BMRA )  in  their  final  report  recommended 
that  several  courses  be  conducted  that  are  tailored  to  the 
experience  level  of  the  source  selection  participants.  The 
concept  of  tailored  courses  seems  appropriate  to  ensure  the 
proper  level  of  training  is  received  by  all  source  selection 
participants.  BMRA  recommends  four  levels  of  training 
and/or  briefing  but,  initially,  two  levels  may  be  sufficient 
to  provide  source  selection  participants  with  the  necessary 
training.  The  courses  could  be  stuctured  as  follows:  a 
five  day  short  course  covering  general  source  selection 
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topics  common  to  all  AFSC  product  divisions,  and  a  one  day 
briefing/refresher  course  conducted  by  the  individual 
product  divisions  covering  selection  specific  topics  and  any 
changes  or  updates  to  the  source  selection  process. 

Recommendations .  The  following  recommendations  outline 
a  suggested  source  selection  training  course  format  based  on 
the  survey  responses  and  comments. 

1.  Two  levels  of  source  selection  training  should 
be  established,  a  general  course  covering  the  entire  source 
selection  process  for  first  time  source  selection 
participants,  and  a  short  one  day  course  prior  to  a  source 
selection  for  experienced  source  selection  participants. 

2.  The  general  source  selection  course  should 
last  approximately  five  days  and  cover  a  wide  variety  of 
topics  from  r lanning  to  contract  evaluation,  to  contract 
award.  The  course  should  be  common  to  all  AFSC  product 
divisions  and  might  best  be  taught  by  a  traveling 
instruction  team.  The  instructors  could  provide  the  course 
on  a  recurring  or  "as  needed"  basis.  Air  Force  Systems 
Command,  AFIT,  or  private  contractor  personnel  might  be 
considered  as  possible  instructors  for  the  course. 

3.  The  short  course  for  experienced  source 
selection  participants  should  last  approximately  one  day  and 
provide  the  attendees  with  specific  information  on  the 
pending  source  selection.  Any  policy  changes  or  updates  to 
the  source  selection  process  could  be  covered,  as  well  as  a 
quick  refresher  on  criteria,  standards,  preparation  of 
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Deficiency  Reports  and  Clarification  Requests,  etc. 
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Area  for  Additional  Research 

A  pilot  course  as  outlined  above  could  be  developed 
prior  to  an  upcoming  source  selection.  Personnel  scheduled 
to  participate  in  the  source  selection  could  be  solicited  to 
attend  the  pilot  course  on  a  volunteer  basis.  After  the 
source  selection  is  completed,  the  participants  (both 
course  attendees  and  non-attendees)  could  be  surveyed  to 
determine  the  benefit  of  the  course. 
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beneficial  to  future  source  selection  efforts. 
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return  it  in  the  enclosed  envelope  within  7  days  of  receipt.  Your 
individual  response  will  be  combined  with  other  responses  and  will  not 
be  attributed  to  you  personally. 

3.  Your  responses  to  Captain  Gray's  and  Captain  Hugo's  survey  provided 
very  valuable  information  and  enabled  them  to  complete  an  outstanding 
research  project  which  was  briefed  to  the  Air  Staff.  Once  again,  your 
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Appendix  B  -  Survey  Instrument 


SURVEY  REGARDING  SOURCE  SELECTION 
IN  AIR  FORCE  SYSTEMS  COMMAND 

************************************************************ 
This  survey  will  take  approximately  25  minutes  to 
complete.  The  survey  contains  four  sections.  The  first 
section  will  provide  background  and  experience  information 
which  will  be  used  for  grouping  opinion  data  from  sections 
two,  three,  and  four.  Your  responses  will  greatly  assist  us 
in  our  analysis  of  the  current  feeling  on  the  preparedness 
of  source  selection  members.  Please  feel  free  to  make 
->dditional  comments  as  you  fill  out  the  survey. 

THANK  YOU  FOR  YOUR  TIME  AND  COOPERATION. 
************************************************************ 

Part  1  -  Background  Information 

1.  Military  Rank  or  Civilian  Grade: _ 

2.  Office  Symbol: _ 

3.  AFSC  Product  Division: 

o  ASD  o  AD 

o  ESD  o  Other  (please  specify) 

o  SD 

4.  With  which  acquisition  phase  are  you  most  familiar: 

(may  select  more  than  one  answer) 

o  Concept  Exploration 
o  Demonstration/Validation 
o  Full  Scale  Development 
o  Production 
o  Other  (please  specify) 

5.  With  which  type  of  acquisition  are  you 

o  Aircraft 
o  Armament 
o  Electronics 
o  Space/Missile 
o  Other  (please  specify) 

6.  Years  of  source  selection  experience: 


o 

0-2  years 

o 

10-15  years 

o 

2-5  years 

o 

15-20  years 

o 

5-10  years 

o 

more  than  20  years 
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7.  Number  of  source  selections  in  which  you  have 
participated: 

8.  Number  of  cost  panels  in  which  you  have  been  a  member: 

9.  Number  of  technical  panels  in  which  you  have  been  a 
member : 

10.  Current  Functional  Area:  o  Program/Project  Management 

o  Contracting/Manufacturing 
Mgmt . 

o  Engineering 
o  Configuration  Management 
o  Program  Control/Business 
Mgmt. 

o  Logistics 
o  Test  and  Evaluation 
o  Cost  Analysis 
o  Other  (please  specify) 

11.  Education  (Highest  degree  awarded): 

o  High  School  Diploma  o  Masters  Degree 

o  Associate  Degree  o  Doctorate  Degree 

o  Bachelors  Degree 

Part  II  -  General  Source  Selection 

12.  From  what  source  did  you  get  the  bulk  of  your  source 
selection  knowledge? 

o  Formal  Air  Force  training  courses 
o  On-the-Job  Training  (OJT) 
o  Individual  Study/Correspondence  Course 
o  I  have  no  experience  in  the  source  selection  process 
o  Through  participation  in  previous  source  selections 
o  Other  (please  specify) 

13.  What  formal  Air  Force  training  courses  have  you 
attended  that  included  material  on  source  selection? 

o  None 

o  Systems  200/400 

o  PPM  152  Contract  Administration 
o  Log  224  Logistics  Management 
o  PPM  302  Government  Contract  Law 

o  QMT  345  Quantitative  Technical,  Cost,  and  Price 
Analysis 

o  Other  (please  specify) 


B-2 


14.  Prior  to  participating  in  your  first  source  selection, 
did  you  receive  any  formal  source  selecton  training? 
(If  you  answer  NO,  please  skip  to  question  15) 

o  YES  o  NO 

14A.  What  type  of  training  did  you  receive? 
o  Briefing 

o  Self-paced  course  or  presentation  (sound-on-slide) 
o  Formal  course  -  Please  specify 
o  Other  (please  specify) 

14B.  Who  provided  the  training  you  received? 

o  Product  Division 

o  Individual  office  or  System  Proram  Office  (SPO) 
o  Source  Selection 
o  Other  (please  specify) 

14C.  How  long  did  your  training  last? 

o  Less  than  one  day 
o  1-2  days 
o  3-5  days 
X)  1  week 
o  2  weeks 

o  Other  (please  specify) 

15.  (If  you  answered  yes  to  question  14,  skip  to  question 
16)  Did  you  receive  any  source  selection  training 
prior  to  subsequent  source  selections  (If  you  answer 
NO,  skip  to  question  16). 

o  YES  o  NO 

15A.  What  type  of  training  did  you  receive? 
o  Briefing 

o  Self-paced  course  or  presentation  (sound-on-slide) 
o  Formal  course 
o  Other  (please  specify) 

15B.  Who  provided  the  training  you  received? 

o  Product  Division 

o  Individual  office  or  System  Program  Office  (SPO) 
o  Source  Selection 
o  Other  (please  specify) 
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15C.  How  long  was  the  training? 

o  Less  than  one  day 
o  1-2  days 
o  3-5  days 
o  1  week 
o  2  weeks 

o  Other  (please  specify) 

16.  Would  you  attend  a  source  selection  training  course 
if  one  was  available? 

o  YES  o  NO 

17.  The  content  of  the  source  selection  training  course 
should  be: 

o  instructions  on  the  Request  for  Proposal  (RFP) 
o  instructions  on  source  selection  forms 
o  general  source  selection  procedures 
o  instructions  on  evaluating  contractors  proposals 
o  other  (please  specify) 

18.  The  length  of  the  source  selection  course  should  be: 

o  Less  than  a  day 
o  1-2  days 
o  3-5  days 
o  1  week 
o  2  weeks 

o  Other  (please  specify) 

19.  The  format  for  any  formal  course  offered  should  be: 

o  Selection  specific  (short  course  prior  to  specific 

source  selection) 

o  General  (i.e.  AFIT  or  AFSC  short  course  covering 
general  aspects  and  procedures) 
o  Video  or  sound-on-slide  presentation  (at  local  base) 
o  Other  (please  explain) 

20.  The  source  selection  course  should  be  given  by: 

o  Product  Division 
o  Individual  office  or  SPO 
o  Source  Selection 
o  Other  (please  specify) 
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21.  If  formal  training  was  received,  please  complete  this 
item.  Formal  training  adequately  prepared  me  for  my  source 
selection  responsibilities.  (circle  appropriate  number) 

- 1 - 2 - 3 - 4 - 5 - 

strongly  disagree  neither  agree  agree  strongly 

disagree  nor  disagree  agree 

22.  What  was  the  biggest  problem  that  you  encountered  as  a 
source  selection  member? 

o  Not  aware  of  the  source  selection  procedures 
o  Unfamiliar  with  interpreting  and  filling  out  forms 
o  Quality  of  information  provided  by  the  contractor 
o  Time  restrictions 

o  Inexperience  of  source  selection  personnel 
o  Not  familiar  with  the  RFP 
o  Other  (please  specify) 

Part  III  -  Cost  Panel  (Source  Selection) 

INSTRUCTIONS:  Answer  the  questions  in  this  section  if  you 

have  participated  in  a  source  selection  as  a  cost  panel 
member . 

23.  How  were  you  selected  for  your  source  selection 
responsibilities  as  a  cost  panel  member? 

o  past  experience 

o  education  level  or  training  received 
o  expertise  in  specific  area 
o  random  selection 
o  only  one  available  at  the  time 
o  part  of  job  responsibility 
o  other  (please  specify) 

24.  If  you  selected  individuals  for  a  source  selection,  how 
did  you  determine  the  members  on  the  cost  panel? 

o  cost  background 
o  education  level 
o  training  received 
o  recommendation  by  others 
o  randomly 

o  other  (please  specify) 

25.  How  prepared  do  you  feel  you  were  for  your 
responsibilities  on  the  cost  panel?  (circle  appropriate 
number ) 


- 1 - 2 - 3 - 4 - 

unprepared  somewhat  neither  prepared  somewhat 
unprepared  nor  unprepare  prepared 


well 

prepared 


26.  How  prepared  do  you  feel  other  members  were  for  their 

responsibilities  on  the  cost  panel?  (circle  appropriate 
number ) 


- 1 - 2 - 3 - 4 - 5 - 

unprepared  somewhat  neither  prepared  somewhat  well 

unprepare  nor  unprepared  prepared  prepared 

27.  Would  additional  training  in  the  area  of  cost  or  cost 

analysis  have  been  beneficial?  (If  you  answer  no,  skip 
to  question  28) 

o  YES  o  NO 


27A.  In  what  areas  concerning  cost  or  cost  analysis 
would  additional  training  be  beneficial? 

o  Source  Selection  cost  considerations  (realism, 
reasonableness,  accuracy,  etc.) 
o  Life  cycle  cost  analysis 
o  Cost  estimating 
o  Sensitivity  analysis 
o  Risk  analysis 
o  Other  (please  specify) 

27B .  The  format  for  the  additional  training  should  be: 

o  short  course  prior  to  specific  source 
selection 

o  Self-paced  course  (sound-on-slide,  video  etc.) 
o  Formal  course 
o  Other  (please  specify) 

27C.  The  additional  training  should  be  given  by: 


o  Source  Selection 
o  Individual  office  or  SPO 
o  Product  Division 
o  Other  (please  specify) 

27D.  The  additional  training  should  be: 

o  Less  than  a  day 
o  1-2  days 
o  3-5  days 
o  1  week 
o  2  weeks 

o  Other  (please  specify) 
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28.  What  was  the  biggest  problem  that  you  encountered  as 
source  selection  member  on  a  cost  panel? 

o  Not  aware  of  source  selection  procedures 
o  Unfamiliar  with  interpreting  and  filling  out  forms 
o  Quality  of  information  provided  by  the  contractor 
o  Time  restrictions 

o  Inexperience  of  source  selection  personnel 
o  Not  familiar  with  RFP 
o  Other  (please  specify) 


Part  IV  -  Technical  Panel  (Source  Selection) 

INSTRUCTIONS:  Answer  the  questions  in  this  section  if  you 

have  participated  as  a  technical  panel  member. 

29.  How  were  you  selected  for  your  source  selection 
responsibilities  as  a  technical  panel  member?  (mark  all 
that  apply) 

o  past  experience 
o  education  level 
o  training  received 
o  expertise  in  specific  area 
o  random  selection 
'o  only  one  available  at  the  time 
o  part  of  job  responsibility 
o  other  (please  specify) 

30.  How  do  you  selected  individuals  for  membership  on  a 
technical  panel?  (If  you  do  not  select  members  for 
a  technical  panel,  skip  to  the  next  question) 

o  technical  background 
o  education  level 
o  training  received 
o  recommendation  by  others 
o  randomly 

o  other  (please  specify) 

31.  How  do  you  feel  individuals  should  be  selected  for 
membership  on  a  technical  panel? 

o  technical  background 
o  education  level 
o  training  received 
o  recommendation  by  others 
o  randomly 

o  other  (please  specify) 
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32.  How  prepared  do  you  feel  you  were  for  your 
responsibilities  on  the  technical  panel? 

- ! - 2 - 3 - 4 - 5 - 

unprepared  somewhat  neither  prepared  somewhat  well 

unprepared  nor  unprepared  prepared  prepared 

33.  How  prepared  do  you  feel  other  members  were  for  their 
responsibilities  on  the  technical  panel? 

- 1 - 2 - 3 - 4 - 5 - 

unprepared  somewhat  neither  prepared  somewhat  well 

unprepared  nor  unprepared  prepared  prepared 

34.  What  do  you  feel  are  major  problem  areas  in  the 
technical  evaluation  of  contract  proposals? 

o  Development  of  the  standards 

o  Interpretation  of  the  contractor's  proposals 
o  Competence  of  the  participants 

o  Unfamiliarity  with  the  technical  panel  procedures 
o  Administrative  problems  (filling  out  forms,  etc.) 
o  There  are  no  problems  (or  only  minor  ones)  in  the 
technical  area 

35.  How  beneficial  would  a  formal  course  be  in  providing 
the  required  source  selection  background  information 
(i.e.  explaining  the  source  selection  process  and 
specific  procedures ) ? 

- 1 - 2 - 3 - 4 - 5 - 

Not  beneficial - Somewhat  beneficial - Very  beneficial 

36.  How  beneficial  would  a  contractor  presentation  (plus 
question  and  answer  session)  be  in  shortening  the 
technical  evaluation  process  or  resolving  proposal 
deficiencies? 

- 1 - 2 - 3 - 4 - 5 - 

Not  beneficial - Somewhat  beneficial - Very  beneficial 

37.  How  beneficial  would  a  formal  course  be  in  providing 
the  required  specific  technical  background  prior  to  a 
technical  panel  review? 

- 1 - 2 - 3 - 4 - 5 - 

Not  beneficial - Somewhat  beneficial - Very  beneficial 

38.  Would  the  technical  panel  or  source  selection  schedule 
allow  for  any  courses  to  be  conducted? 

o  YES  o  NO 
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If  yes,  the  optimum  length  for  a  source  selection 
course  would  be 

o  One  day 
o  Two  days 
o  One  week 
o  Two  weeks 

o  Other  (please  specify) 

V.  Please  provide  any  additional  comments  that  you  may 
have : 


THANK  YOU  FOR  YOUR  COOPERATION 
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The  comments  listed  in  this  appendix  reflect  the  views  cf 
each  survey  participant  as  written  and  do  not  necessarily 
represent  the  views  of  the  author  or  the  position  of  the 
Air  Force  Institute  of  Technology  or  the  United  States  Air 

Force . 

Part  I  Survey  Questions 

Question  4:  With  which  acquisition  phase  are  ycu  most 
familiar? 

Major:  Program  Control 

GS-12 :  FNS 

GM-15:  Modifications  (both  major  and  minor) 

GM-14:  Contractor  Logistics  Support 

GS-13:  C3I 
GS-12:  Deployment 

GS-11:  RFP  preparation,  source  selection 

Lt  Col:  A&CO/Deployment 

Question  5:  With  which  type  of  acquisition  are  you  most 
familiar? 

GS-17 :  Avionics  Specifically 

1  LT:  Engines 
1  LT:  Simulators 
GS-12:  Training  systems,  radars 

1  LT:  Simulators 

GS-12:  Support  equipment 

Major:  Simulators  and  training  systems 

Lt  Col:  Electro-Optics  R&D 
GM-13:  Support  equipment 

GS-13:  Avionics 

GM-15:  Technology  demonstration 

Question  10:  Current  Functional  Area: 

Captain:  Legal 

Lt  Col:  Staff  Judge  Advocate 

Lt  Col :  Legal 

2  LT:  Executive  Officer 

PART  II  Survey  Questions 

Question  12:  From  what  source  did  you  get  the  bulk  of  your 
source  selection  knowledge? 

GM-13:  AFR  70-15  and  practical  experience 
GS-12:  School  of  hard  knocks 


Question  13:  What  formal  Air  Force  training  courses  have 
you  attended  that  included  material  on  source  selection? 

GS-13:  QMT  353,  LOG  225 

GM-13:  Computer  resource  acquisition  course 

GM-13:  DSMC,  Financial  Mgt  for  PM 

Lt  Col:  AFIT  (Master  in  Law  in  Federal  Contract  Law) 
Col:  DSMC 

Lt  Col:  AFIT 
Lt  Col :  DSMC 

Question  14A:  What  type  of  training  did  you  receive? 

GM-15:  Read  AFR  70-15 

Question  14B:  Who  provided  the  training  you  received? 

GM-14:  AFSC 

Captain:  Army  JAG  School 

GM-13:  Combination  of  Product  Division  and  Procurement 
GM-13:  Contracting  Officer 

i 

Question  14C:  How  long  did  your  training  last? 

GM-14:  4  weeks 

Question  15A:  What  type  of  training  did  you  receive? 

Captain:  Read  the  regulations 
Captain:  Self-study 
Col:  OJT  and  DSMC 

GM-13:  Self  taught  with  help  from  staff 

Major:  Self  study 

Question  15B:  Who  provided  the  training  you  received? 
Captain:  Self-taught 

Captain:  Staff  Source  Selection  officer  and  try  own 

efforts  at  self-study 
Col :  DSMC 

GM-14:  Cost  analysis  division  staff 
Major:  Self  initiative 

Question  16:  Would  you  attend  a  source  selection  training 
course  if  one  was  available: 

YES  RESPONSES 

GM-14:  More  likely,  would  send  people 
NO  RESPONSES 

Col:  Not  at  this  point  in  my  career 


GM-15:  I  might  send  my  people  if  course  was  meaningful 

GM-14:  Because  now  I  teach  them  myself 

GM-13:  Would  have  previously 

GM-15:  Only  because  I'm  retiring,  otherwise  yes 
Lt  Col :  I 'm  retiring 

Question  17:  The  content  of  the  source  selection  training 
course  should  be: 

GM-13:  All  of  the  above  plus  regulatory/policy 

familiarity 

1  LT:  Plus  how  the  Government  estimates  interfaces 
with  the  RFP  and  its  use  within  the  briefing 

GM-14:  Instruction  on  writing  up  the  evaluations 

Lt  Col:  Tips  on  schedules,  files  organization, 
procedures  for  tracking  CRs  and  DRs 

GM-14:  Developing  evaluation  standards 

Captain:  Mock  source  selection--to  include  evaluation  . 

of  proposals 

Major:  Overview  of  PK  role.  Part  of  source  selection 

training  should  be  review  of  "lessons  learned"  from  other 
source  selections  in  recent  past 

GM-14:  Strategies,  negotiation  skills,  view  from 

contractor's  side,  tricks  of  trade,  etc.  (real  life) 

Question  18:  The  length  of  the  source  selection  course 
should  be: 

Major:  Vary  depending  on  experience  of  individuals-- 
few  hours  for  someone  who  has  gone  through  it  before;  1-2 
days  for  first  timer. 

GM-14:  Self-paced 

Question  19:  The  format  for  any  course  offered  should  be: 

GS-12:  One,  general,  including  cost,  technical,  and 
contractual  aspects.  Second,  specific  to  the  particular 
problem  of  acquisition. 

Captain:  Each  product  division  should  conduct  specific 

training  on  source  selection  within  that  division. 

Major:  Must  be  local,  gear?d  to  product  division 
peculiarities;  must  be  current  and  should  address  specifics 
of  the  individual  program. 

GS-13:  Method  of  asking  CRs/DRs  and  how  to  participate 
in  negotiations 

Lt  Col :  Suggest  general  AFIT/AFSC  2  week  course  then 
short  1-2  day  course  prior  to  specific  source  selection 

Question  20:  The  source  selection  course  should  be  given 
by: 

No  Rank/Grade:  Combination  from  above 

GM-14:  Combination  of  product  division  and  source 


selection 

GM-13:  Seminar  at  each  product  division 

GM-14:  AFIT  (better  overall  perspective) 

Col:  AFIT  with  lectures  from  product  divisions  and 
source  selection  personnel 

GM-15:  Functional  offices  AC/EN/PM,  etc. 

GM-15:  Course  should  contain  generic  source  selection 
guidelines  by  source  selection  and  selection  specific 
information  by  SPO 

GM-14:  Someone  at  product  division  who  is 

knowledgeable  and  has  practical  experience 
GM-13:  AFIT  or  DSMC 

GM-15:  Formal  AFSC  course  given  by  product  division 
GM-15:  Functional  Organization  —  Engineering 

Question  22:  What  was  the  biggest  problem  that  you 
encountered  as  a  source  selection  member? 

GM-15:  Almost  everyone  has  never  been  through  one 
before ! 

GM-14:  Lack  of  specific  formats  required  by  SSA 

Major:  Changes  in  procedures  midstream — numerous  times 

Captain:  Panel  chiefs  continually  changing  formats  and 

philosophies  of  the  formal  write-ups,  thereby  forcing 
workers  to  waste  hours 

GM-14:  Restrictions  on  access  to  contractors 
GM-15:  Lack  of  quality  can  be  both  government  and 
contractor's  fault  (i.e.,  unclear  instructions  on  part  of 
govt,  or  contractor  ignoring  instructions) 

GM-13:  The  quantity  of  ridiculous  procurement 
requirements  made  P/0  Proposals  extensive  review  of  material 
to  determine  that  applicable 

GM-13:  Imprecise  guidance  which  tended  to  evolve 

during  course  of  SS,  thus  inefficient  and  wasteful 

GM-15:  Not  PMPS  personnel  but  SPO  personnel  running 
the  source  selection  (comment  pertaining  to  inexperience  of 
source  selection  personnel) 

GS-13:  General  confusion  and  redundancy  of  effort. 

Not  being  familiar  with  the  RFP  was  true  of  people  outside 
of  the  SPO 

2  LT:  Quality  of  information  provided  by  AFR  70-15 
Col:  Inexperience  of  source  selection  personnel. 
Expecially  those  in  supervisory  positions  that  should  be 
advising  the  younger  members. 

1  LT:  All  are  problems 

GS-13:  Not  being  able  to  see  cost  figures.  Incomplete 

process  when  you  can't  see  all  factors 

GS-13:  Establishing  appropriate  and  consistent 
philosophy  for  evaluation 

GS-13:  Poor  working  conditions  (heat,  crowds,  parking) 
GM-14:  Inexperience  in  establishing  good  evaluation 

criteria/standards 

Lt  Col:  Awareness  of  significance  of  following 


C-4 


procedures  to  ensure  legally  OK  award. 

GS-12:  What  to  evaluate  in  the  proposal  as  opposed  to 

what  was  required  by  the  ITO  (Section  L  of  RFP ) 

Part  IV  Survey  Questions  (Technical) 

Question  29:  How  were  you  selected  for  a  technical  panel? 

Col:  Program  Manager  of  program  in  selection 

Question  30:  How  do  you  select  technical  panel  members? 

Captain:  SS  experience 

GM-13:  Matrix  engineer  to  SPO 

GM-13:  Head  of  project  office  nominates  technical 

personnel 

1  LT:  Whoever  is  available 

SES:  Associated  with  the  program 

Col:  Representatives  of  using  Commands 

Col:  Past  source  selection  experience 

GS-13:  Availability  and  familiarity  with  the  program 

Question  31:  How  should  technical  panel  members  be 
selected? 

Captain:  SS  experience 

GM-13:  Matrix  responsibilities 

SES:  Associated  with  the  program 

Question  34:  Major  problem  areas  in  the  technical 
evaluation  of  contract  proposals: 

GM-13:  Personnel  with  other  responsibilities  not 

giving  SS  enough  of  their  time. 

GS-12:  Evaluation  against  the  ITO  requirements 

GS-13:  Unclear  definition  of  ratings  and  risk 

SES:  Requesting  and  reviewing  too  much  irrelevent 
material 

Lt  Col:  Keeping  panel  ratings  consistent  across  all 
offerors 

Lt  Col:  Awareness  of  the  need  to  follow  procedures 
precisely  to  avoid  protests 
1  LT:  No  real  problems 

Lt  Col:  Effect  of  strong  personalities  stifling 
dissenting  opinions 

GS-13:  Usually  cannot  fully  define  contractors' 
performance  requirements 

GS-13:  Not  seeing  cost  data  and  the  formality  of  the 
whole  thing 

Col:  We  spend  our  time  evaluating  a  proposal  that  we 
don't  put  on  contract. 
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Question  39:  Length  of  source  selection  course: 

GS-13:  Standard  video  training  on  specific  aspects 

available  in  source  selection  if  needed 

1  LT:  One  day  for  background  on  process,  one  day  for 
technical  background 

Lt  Col :  3-5  days 

Col:  1/2  day 

Part  V.  Other  Comments 

Captain:  More  formal  training  would  be  helpful. 

Source  selection  teams  are  rarely  the  same  people.  Always 
some  new  members  on  each  team.  Experience  usually  varies 
greatly.  My  training  method  largely  consisted  of  gathering 
and  reading  all  the  pertinent  regulations  I  could  find. 

GM-13:  Because  competitive  acquisitions  are 

accomplished  in  an  environment  where  losing  contractors  may 
sustain  a  significant  loss,  we  must  do  all  that  is  possible 
to  make  sure  all  qualified  offerors  are  treated  equitably. 
This  condition,  in  practice,  limits  the  communication  with 
industry  during  the  source  selection  process.  Because  of 
this,  I  believe  it  is  very  important  to  have  all  source 
selection  team  members,  technical  and  cost  panels,  familiar 
with -the  acquisition  process  prior  to  their  assignment. 

It  is  not  an  environment  where  the  contractor  can  be 
educating  government  evaluators  who  are  new  to  the 
acquisition  process  and  don't  know  the  government's  role  in 
that  process. 

At  SD,  cost  panel  members  are,  for  the  most  part, 
familiar  with  methods  of  determining  fair  and  reasonable 
pricing.  It  is  not  that  different  applying  these  methods  in 
a  competitive  or  non-competitive  acquisition. 

The  technical  members,  on  the  other  hand,  must  have  the 
knowledge  of  the  product  being  acquired  sufficiently 
mastered  to  enable  evaluation  of  differing  approaches  taken 
by  the  competitors  to  supply  the  product.  It  is  not 
appropriate  for  the  winner  of  the  competition  to  be  selected 
on  the  basis  of  his  marketing  or  educational  abilities. 

GS-14:  The  biggest  problem  I  see  is  that  each  source 
selection  seems  to  start  from  the  beginning  in  terms  of 
personnel  knowledge.  No  one  seems  to  know  just  what  needs 
to  be  done.  Most  of  the  people  involved  are  doing  it  for 
the  first  time. 

Recommend  that  a  source  selection  core  team  be 
established  at  each  product  division  to  oversee  all  source 
selections  and  to  provide  training  and  advice  to  those  doing 
the  work. 

GM-15:  ASD/ACC  is  responsible  for  chairing  all  70-15 
source  selection  cost  panels  where  ASD/CC  or  above  is  the 
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SSA.  We  also  generally  provide  the  majority  of  the  cost 
analysts  assigned  to  the  panel.  ASD/ALT  is  responsible  for 
the  O&S  portion  of  the  cost  panel  evaluation. 

ACC  has  solved  the  ongoing  problem  of  lack  of  source 
selection  experience  by  insuring  that  all  cost  panel 
chairmen  (generally  GS-12s  to  GM-14s)  have  participated  in 
one  or  more  source  selections  as  a  cost  panel  member  before 
assignment  as  panel  chairman.  In  addition,  we  routinely 
assign  trainee  cost  analysts  (GS-11  and  below)  under 
experienced  cost  panel  chairmen  as  a  part  of  their  formal 
training  (OJT)  in  the  structured  "comptroller  training 
program."  Successful  participation  in  this  training  program 
is  a  necessary  element  in  their  promotion  from  GS-7  to  GS- 
12.  As  the  designated  "AC  source  selection  focal  point"  I 
am  extremely  interested  in  improving  the  process  and  look 
forward  to  your  report. 

GS-12:  Source  selections  at  SD  are  bound  by  too  many 

unnecessary  administrative  details  (e.g.,  preparation 
instructions  which  are  too  detailed  for  contents  of  decision 
document).  The  requirement  for  PMC  and  JA  review  of  all 
request  for  BAFO's  also  is  very  time  consuming. 

1  LT:  I  think  the  whole  source  selection  process 
should  be  reviewed  with  those  who  have  been  on  the  cost  and 
technical  teams  as  well  as  those  who  have  received  these 
briefings.  Is  the  information  contained  within  the  briefing 
enough,  too  much,  or  just  right?  I  feel  a  review  by  a  group 
consisting  of  the  above  people  should  be  accomplished  at 
least  every  five  years. 

GM-14:  A  short  cost  panel  course  for  "neophytes"  that 
walked  them  through  a  mock  proposal  would  be  helpful.  The 
course  could  consist  of  various  problems  encountered  in 
proposal  analysis  (e.g.,  contractor  spreads  production  $  by 
expenditures  rather  than  fiscal  year  buy,  cost  for  WBS  level 
III  elements  don't  add  up  to  the  level  II  elements,  etc.) 

Major:  Technical  team  members  should  be  made  aware  of 
the  requirements  of  the  cost  team — i.e.,  cost  team  can't 
perform  risk  assessment  without  technical  input.  Source 
selection  members  should  be  picked  far  enough  in  advance  to 
allow  them  to  become  familiar  with  RFP.  Members  who  are  not 
directly  involved  with  SPO  should  be  briefed  on  the  program 
objectives,  the  RFP,  expected  proposal  structure — 
options/blocks,  etc. 

Formats/reports/briefing  are  all  fairly  fluid  and  every 
effort  should  be  made  to  have  the  latest  info  available  to 
SS  team  members. 

GM-13:  The  most  problematic  areas  in  a  source 
selection  are  the  source  selection  plan  (what  are  the 
criteria),  the  problem  preparation  instructions  (What  do  you 


want  to  know  and  evaluate)  and  the  evaluation  guide  (how  is 
the  data  evaluated).  Once  these  three  general  areas  are 
resolved,  source  selection  is  no  more  difficult  than  a 
regular  acquisition.  AFR  70-15  sets  out  the  process  so  any 
training  should  be  concentrated  on  these  three  areas. 

GM-14:  Lack  of  experience  on  every  source  selection  is 

a  severe  handicap — and  will  continue  to  be,  as  no  one  wants 
to  do  them  twice  (and  turnover  ensures  this).  A  short 
course  of  any  kind,  covering  both  procedures  and  practical 
aspects  would  help  at  least  normalize  the  perception  of 
what's  to  be  done,  and  probably  enhance  the  process. 

GM-13:  Training  should  be  provided  to  all  personnel 
working  in  a  position  that  may  be  called  upon  to  work  on  a 
source  selection  board.  There  is  not  enough  time  to 
schedule  people  for  training  and  provide  the  training  after 
they  are  picked  for  a  source  selection  board. 

Technical  personnel  should  already  have  the  technical 
training  and  experience  in  their  respective  areas  of 
expertise  prior  to  being  selected  for  source  selection 
boards . 


GM-13:  The  best  way  to  have  meaningful  and  fast  source 
selections  is  to  reduce  the  RFP  to  a  short  concise 
requirement.  The  majority  of  the  RFP  is  now 
legal/contractual  garbage. 

We  could  never  operate  this  way  during  a  wartime 
emergency  and  should  not  operate  this  way  because  of 
personnel  without  jobs  sitting  around  levying  rights. 

GS-12:  There  should  be  (a)  a  video  for  basics,  (b) 
instructions  in  tools  and  procedures  to  follow  (i.e.,  form, 
schedules,  lessons  learned,  shortcuts),  (c)  specifics  for 
the  acquisition  in  whatever  depth  seems  appropriate, 
including  identification  of  goals,  team  players,  anticipated 
problems,  milestones,  review  process. 

GS-12:  I  have  been  on  three  source  selection  panels 

and  am  currently  involved  in  my  fourth.  Each  one  has  their 
own  way  of  doing  things,  such  as  types  of  questions  to  ask 
on  a  CR  or  DR.  This  creates  a  lot  of  hassle  and  takes  more 
time  to  learn  how  to  do  things  "their"  way. 

Some  of  the  SS  board  members  do  not  understand 
configuration  and,  therefore,  have  a  difficulty 
understanding  what  the  contractor  should  have  provided.  It 
then  turns  out  to  be  a  training  session  because  they  do  not 
feel  comfortable  with  our  decisions. 

2  LT:  ASD/PMPS  should  provide  the  training  and 
supplies  to  SSEB  and  admin,  officers  to  facilitate  the 
source  selection.  Should  help  set  up  files,  CD/DR  process, 
reports,  etc ...  instead  of  saying  "look  it  up  in  AFR  70-15"  — 
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which  is  a  useless  waste  of  paper. 

Also,  let's  start  putting  this  stuff  on  disks.  One 
offeror  offered  to  put  their  cost  information  on  disks  and 
provide  a  computer  and  printer  to  read  it.  ASD/PM  said  no. 
Too  much  time  is  wasted  destroying  200+  bags  of  destruct  and 
filing  200  books  into  files  that  no  one  will  ever  look  at. 

The  SSAC  is  a  waste.  Either  get  them  in  for  two  weeks 
to  write  an  analysis  or  write  them  out  of  AFR  70-15. 

Bigshots  fly  in  for  one  day.  Tell  us  we're  wrong  and  to 
rewrite  our  findings,  and  then  blab  the  information  all  over 
Washington. 

The  source  selection  facility  is  too  small  and  the 
phones  are  obsolete.  They  provide  little  or  no  support.  It 
is  insane  for  people  to  bring  up  their  own  supplies  and 
equipment  and  move  it  in  and  out  when  PMPS  could  maintain 
it.  They  also  should  provide  administrative  advice  if  not 
admin,  officers  outright. 

GM-15:  AFSC  should  develop  a  formal  training  course 

for  first  time  source  selection  participants.  The  product 
division  should  develop  an  abbreviated  course  or  briefing  to 
be  given  before  each  source  selection  as  a  refresher  course. 

Captain:  I  am  a  newly  appointed  source  selection 

officer.  Also,  I've  only  been  in  my  current  job  six  months. 
To  date,  I  have  not  received  any  formal  training  nor  have  I 
participated  in  any  source  selections.  However,  I  do  have 
two  in  the  near  future.  I  strongly  recommend  that  all  new 
or  inexperienced  source  selection  officers  receive  formal 
training. 

Lt  Col:  AFSC  is  the  most  non-professional  command  in 
the  USAF.  It  hires  engineers  to  do  the  business  manager 
work  with  no  enroute  pipeline  training  on  the  acquisition 
process  or  specifics  thereto.  It  is  much  like  the  USAF 
giving  a  guy  an  F-15  to  fly  around.  If  he  likes  it  and 
doesn't  crash  and  funds  permit,  he  is  sent  to  UPT.  I  wonder 
if  management  cares.  SPOs  are  manned  half-assed  or 
otherwise  indifferently  except  for  the  truly  hi-viz  projects 
like  B-l.  The  taxpayers  and  the  USAF  are  ill-served  by 
AFSC.  Can't  run  this  business  like  a  tactical  unit, 
especially  without  pipeline  training. 

Lt  Col:  Training  is  badly  needed — for  evaluators,  for 
administrative  people,  for  area  chiefs,  item  captains. 

Ideal  would  be  to  train  for  individual  source  selection,  but 
that  would  be  impractical.  Might  have  general  training 
spread  over  several  days;  1st  day  general  procedures  and 
product  division  policies;  remaining  days  in  smaller 
sessions  specifically  for  details  of  administration,  area 
panel  operations,  etc.  However,  anything  you  do  will  be 
better  than  what  exists. 


Captain:  One  of  the  problems  that  exists  on  every 

source  selection  I 've  seen  is  a  lack  of  understanding  by  the 
program  office  of  the  functions  and  responsibilities  of  the 
contracting  office  and  cost  panel  proposal  evaluation  team. 
Program  offices  seek  to  dictate  schedules  without  any 
insight  into  what  reviews  and  steps  are  required  prior  to 
contract  award.  They  seek  to  dictate  what  contract  clauses 
are  to  be  included  in  a  contract,  without  any  insight  into 
FAR  requirements.  Program  offices  also  try  to  dictate  how 
line  items  will  be  structured  without  any  knowledge  of 
funding  constraints,  etc.  All  the  above  cause  buying 
personnel  to  spend  inordinate  amounts  of  time  explaining  and 
arguing  over  items  required  by  FAR.  Because  program 
managers  misunderstand  their  roles  in  the  source  selection 
process,  they  tend  to  concentrate  on  choosing  a  successful 
offeror  rather  than  presenting  facts  to  the  SSA. 

Based  on  the  above,  I  recommend  that  a  specific  course 
be  structured  for  program  managers  alone,  who  serves  as  SSEB 
chairpersons.  The  program  manager's  roles  should  be  clearly 
presented  as  one  of  managing  the  technical  evaluation  only. 
They  should  understand  that  contracting  personnel  are 
responsible  for  acquisition  decisions  affecting  the  RFP, 
model  contract,  and  award  schedules.  A  program  manager 
(SSEB  chairperson)  course  should  include  the  following: 

-  Program  manager  roles  and  responsibilities 

-  SSEB  organization 

-  Information  control — efficient  flow  up  and  down  of 
source  selection  information 

-  Source  selection  documentation  requirements  (forms): 
e.g.,  factors  summaries,  item  summaries,  CRs/DRs,  PFNs,  area 
summaries 

-  Documentation  required  in  technical  evaluations  as 
proper  inputs  into  the  cost  panel  evaluation,  e.g.,  hours, 
materials,  etc. 

-  Proper  review  of  CRs  and  screening  of  DRs,  e.g.,  DRs 
must  track  to  factors  and  standards  and  to  a  RFP 
requirement.  Too  often  a  program  manager  reinforces  SSEB 
members  when  they  cite  deficiencies  that  don't  track  to  a 
RFP  requirement  but  to  something  they  would  have  liked  to 
have  been  a  requirement.  Also,  the  program  manager  should 
understand  that  all  offerors  are  treated  the  same.  DRs 
should  not  be  given  to  only  one  offeror  if  they  apply  to 
others  and  vice  versa. 

-  Competitive  range — what  it  is  and  why  it  is  used. 
Program  managers  at  ESD  seem  to  think  a  competitive  range 
determination  excluding  offerors  is  only  to  be  made  if  it 
supports  the  program  schedule.  They  don't  understand  the 
legal  and  monetary  ramifications  of  including  or  excluding 
offerors  erroneously. 

-  Schedule  management — program  managers  should  be  fully 
aware  of  all  source  selection  briefing  requirements,  and 
reviews.  Because  it  is  the  procurement  community  which 
normally  takes  all  schedule  heat,  program  office  input  into 


schedule  should  be  merely  a  statement  of  user  needs  and 
delay  impacts. 

A  program  manager  (SSEB  chairperson)  training  session 
should  be  given  within  each  product  division  prior  to 
conducting  the  SSEB  training  I  recommended  earlier.  The 
program  manager  could  then  be  tasked  with  organizing  the 
SSEB  and  performing  all  his  responsibilities  at  the  mock 
session  under  supervision  of  trained  instructors.  This 
should  be  about  a  three  day  training  course. 

GM-15:  Formal  source  selection  courses  are  needed. 

Suggest  contacting  the  source  selection  secretariats  at  AFSC 
product  division  for  h^lp  in  structuring  such  a  course. 

Right  now  there  is  too  much  trial  and  error  learning  of  the 
most  important  aspects  of  our  acquisition  business-source 
selection . 

GM-14:  Our  problems  at  BMO  in  the  training  area  stem 

from  the  lack  of  resources.  I  have  been  the  source 
selection  officer  for  the  last  six  years  (and  assistant  SSO 
for  three  years  before  that)  but  on  a  "catch-as-catch-can" 
basis . 

GS-12:  Preparation  for  source  selection  should  start 

when  the  Statement  of  Work  and  the  ITO  are  drafted. 

Sequence  of  events: 

1.  PMD  requirements  identified 

2.  SOW  identified  and  written  to  include  PMD 
requirements 

3.  Draft  Section  "L"  and  "M"  of  RFP 

4.  Evaluation  of  requirements  -  Section  "M"  and  check 
against  Section  "L" 

5.  Evaluate  Section  "L"  of  RFP  (draft  same  time  as 
section  "M" ) 

6.  Develop  Standards  and  Factors 

7.  Develop  checklist  for  each  factor 

8.  Review  Source  Selection  Plan  and  procedures 

9.  Conduct  training  for  source  selection  using: 

— lessons  learned  from  other  source  selections 

— video  tape  of  members  in  past  source  selections 
explaining  strategy  and  methods 
— practice  writing  and  using:  SOW,  RFP,  ITO,  CDRL, 
Standards/Factors,  Evaluation  forms,  ASD  forms 
356,  357,  359. 

2  LT:  Source  Selection  is  a  very  demanding  time  for 
the  program  office.  Each  member  of  the  technical/cost  team 
must  be  committed  100+%  to  the  program's  success  and  not 
exterior  activities.  The  chain  of  command  in  a  source 
selection  must  be  followed  exactly.  The  SSAC  chairman 
reports  to  the  SSA;  the  SSEB  chairman  reports  to  the  SSAC 
chairman;  and  the  Source  Selection  Executive  Officer  (SSEO) 
reports  to  the  SSAC  chairman  and  SSEB  chairman.  All  of  the 


iUi:i.  i  n  i  st  rat  ive  aspects  of  the  SS  are  the  responsibi  1  i  ty  of 
the  SSEO  and  he/she  must  have  the  proper  authority  to  carry 
cut  his/her  assigned  tasks.  However,  rank  often  restricts 
the  SSEO  to  accomplish  these  "firefighting"  tasks.  The  SSAC 
chairman  and  SSEB  chairman  should  allow  the  SSEO  the  freedom 
to  use  ill  of  his/her  resources  to  accomplish  the  SSEO 
mission:  to  effectively  run  the  source  selection. 

GS-13:  It  is  difficult  to  do  or  prepare  for  a  joint 

program,  DOD  major  program,  with  off-site  evaluation  sites! 
My  experience  is  two  source  selections  on  INEWS  --  a  joint 
(AF-USN)  DOD  major  program  entering  its  Dem/Val  phase. 

Major:  A  formal  course  should  be  offered  in  two 
distinct  phases.  One  phase  prior  to  the  preparation  of  the 
ITO,  SS  Plan,  and  SS  standards.  The  second  phase  would 
occur  just  prior  to  the  start  of  SS  and  would  include  all  SS 
evaluators  (using  Command  etc.).  The  ideal  candidate  for 
the  responsibility  would  be  PMPS  (at  ASD)  or  equivalent 
office. 

Lt  Col :  My  source  selection  experience  has  been  in  the 
lab,  not  at  ASD.  We  didn't  put  people  on  the  technical 
panels  who  wouldn't  understand  the  technical  issues  —  they 
came  prepared.  Dialog  with  the  contractor  about  his 
proposal  (open,  one-on-one)  would  have  been  great,  but  I 
don't  know  how  we  cculd  have  pulled  it  off.  Consider  short 
formal  courses  on  selected  topics  during  evaluation  in  case 
proposals  go  far  afield  from  available  expertise. 

GM-13:  My  personal  experience  is  that  the  source 
selection  process  is  too  compressed  and  does  not  allow  for 
proper  administrative  procedures  to  be  followed.  Also, 
panel  members  are  not  100%  committed  to  the  source  selection 
activities,  and  the  process  is  spread  out.  The  combination 
of  these  two  factors  appear  to  create  most  problems. 

GS-13:  1.  Ninety-five  percent  of  technical 

evaluations  are  "not  used",  i.e.,  only  those  few  things  that 
"poke-thru"  the  OK/Green  (Ex.  HI/LO  risk  items)  get  onto 
final  briefing  forms/item  and  area  write-ups.  Too  bad  that 
source  selection  activity  is  not  focused  on  providing  this 
5%. 

2.  Local  accepted  norms  of  "chartmanship"  drives  (through 
backfitting)  what  gets  reported  --  i.e.,  the  linkage  of 
ratings  and  risks,  and  strengths/weaknesses. 

The  above  two  items  restrict  the  final  evaluation  verbage 
and  charts.  This  should  be  included  in  the  Regs,  and 
thoroughly  explained  to  technical  members  so  they  understand 
and  can  focus  their  efforts.  Video  tapes  readily  available 
to  show  short  subjects  would  be  nice.  Source  selection 
facility  needs  more  typing  help,  more  copiers,  and  a  way  to 


make  briefing  charts  etc.  Maybe  it's  time  to  go  mostly 
electronic  instead  of  the  massive  amount  of  paperwork  (4  to 
5  thousand  sheets  for  technical  alone  on  my  last  source 
selection).  One  of  our  bidders  offered  to  provide  cost 
volume  on  floppy  disk  and  provide  computer  and  printer  to 
read  and  make  copies.  The  contract  folks  (PM)  declined  this 
offer . 

GM-14:  The  general  feeling  that  I  have  heard  at  ASD 

over  the  past  15-20  years  is  that  the  source  selection 
process  is  too  short  for  what  is  required  of  the  individual. 
I  believe  the  problem  is  that  the  individual  is  not  prepared 
(trained)  for  the  specific  SS.  Each  are  unique  to  some 
degree.  Let's  prepare  our  people. 

SES:  The  keys  to  a  successful  source  selection  are: 

— focus  on  the  key  issues  (not  all  SOW/Spec 
requirements ) . 

— use  knowledgeable  personnel  who  have  a  stake  in  the 
outcome  (supplement  by  experts  as  required) 

— keep  the  proposals  short 

— visit  the  offerors  as  part  of  the  process. 

GS-12:  Panel  chairpersons  should  be  very  knowledgeable 

program  personnel  and  be  well  informed  on  the  SPE,  ITO  and 
SOW  BEFORE  start  of  standard  writing.  Standards  should  be 
written  before  the  ITO  is  finalized.  SOW  should  state  work 
effort  but  not  specifically  "how".  Then  proposed  "how”  can 
be  evaluated  and  the  better  approach  selected.  SSEB 
Chairman,  Panel  Chairmen,  and  Admin  should  be  required  to 
report  to  the  SS  facility  for  a  minimum  of  one  week  before 
evaluators  report  or  proposals  are  received  to  set  up,  go 
over  procedures,  and  become  familiar  with  forms  and  SSEB 
requirements.  ASD  should  cut  out  the  quick-look  briefing 
and  as  received  briefing.  Eliminate  tht  SSAC  or  mandate 
that  they  do  the  comparison.  Limit  the  number  of  pages  a 
contractor  can  submit  in  response  to  a  Deficiency  Report  or 
Clarification  Report. 

Several  references  within  the  survey  regard  educational 
level.  If  Standards  are  written  well,  educational  level  has 
no  impact  on  the  factor  evaluations.  Item  Captains  and 
Panel  Chairmen  should  have  the  ability  to  write  sound, 
concise,  summary  reports.  Panel  Chairmen  need  to  be  able  to 
communicate  effectively  in  writing  and  in  face-to-face 
negotiations  and  discussions. 

Lt  Col:  The  need  for  strict  adherence  to  procedures 
needs  much  emphasis.  New  laws  can  hold  up  a  contract  for 
months  while  a  protest  is  resolved.  If  people  do  not  follow 
procedures  exactly,  the  agency  might  lose  the  protest  and 
have  to  re-compete.  Programs  cannot  usually  stand  these 
delays.  People  need  to  know  what  has  to  be  done  and  make 
sure  they  do  it  correctly. 
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Col:  The  formal  70-15  approach  of  breaking  evaluations 

down  into  items,  factors,  and  subfactors  causes  evaluators 
to  lose  sight  of  the  big  picture  and  can  lead  to  the  wrong 
decision  since  interaction  between  "pieces"  can  be  the 
critical  item.  The  te-chnical  evaluation  should  be  done  by  a 
small  group  of  experts  using  very  few  (but  critical)  factors 
and  standards.  Each  evaluator  should  read  the  entire 
proposal  to  get  an  integrated  picture.  For  an  example  of 
how  this  can  be  done  in  reality  see  "Increasing  Competition 
Through  Streamlined  Source  Selections"  in  the  May-June  86 
issue  of  DSMC  Program  Manager  magazine. 

GS-13:  Technical  panel  members  for  source  selection 

are  picked  for  their  experience  and  technical  expertise.  No 
one  wants  a  panel  member  that  has  just  been  "trained"  for 
the  source  selection. 

Major:  Because  technical  panel  members  were  selected 

based  on  their  areas  of  expertise,  technical  knowledge  was 
not  a  problem.  Their  most  difficult  problem  was  in  filling 
out  forms  in  proper  formats,  especially  the  Change  Requests 
(CRs).  The  other  serious  problem  was  evaluator  and 
advisors'  supervisors  not  allowing  their  people  to  spend  the 
required  time  evaluating.  Home  office  work  sometimes  put 
the  evaluation  schedule  behind. 

GS-13:  Everyone  should  attend  a  one  day  course 
explaining  the  source  selection  process  in  general  if  they 
have  never  been  on  a  source  selection.  Before  a  specific 
source  selection,  the  SPO  should  give  a  briefing  on  the 
specifics  of  that  source  selection  (review  standards,  dates 
for  write-ups,  review  SS  organization,  and  anything  unique 
to  that  source  selection). 

Col:  In  general,  source  selections  are  too  long  and  go 

into  too  much  technical  depth  since  we  essentially  throw  the 
proposal  away  when  we  are  done  with  the  source  selection. 

The  BMO  has  the  best  approach  I've  found;  I  suggest  you 
contact  them  if  you  haven't  already. 

GS-12:  In  respect  to  the  source  selection  rule  of 

separating  technical  analysis  from  cost  analysis  at  the 
panel  and  panel  member  level,  I  believe  there  is  a  severe 
error  in  procedural  set-up.  The  two  are  linked  in  terms  of 
technical  and  cost  trade-offs  and  the  analysis  regarding  the 
integrity  of  the  propose!!  This  should  be  changed  so 
accounting/financial  and  technical  (i.e.  engineers, 
logistical,  etc.)  can  effectively  work  together  in 
evaluating  the  probability  of  success  given  within  proposed 
costs! 

GS-13:  It's  my  observation  that  those  individuals 

producing  source  selection  policy  and  procedures  are  not  the 


individuals  with  the  kind  of  experience  it  takes  to  best 
accomplish  this.  Many  commanders  (especially  ESD's)  are 
annoyed  at  the  time  required  to  accomplish  source 
selections,  but  instead  of  having  experts  revise  the 
procedures  and  policies,  they  simply  mandate  accelerated 
schedules  and  smaller  proposal  page  counts.  There  are  many 
better  ways  to  improve  the  source  selection  process  but  they 
are  known  by  a  handful  of  middle  managers  who  have 
participated  in  multiple  selections  but  feel  their  ideas 
will  fall  on  deaf  ears. 

GS-11:  Having  a  file  on  base  that's  kept  very  up-to- 

date  with  current  program  managers  '  and  PCOs  '  phone  numbers 
would  save  a  lot  of  time  (or  even  a  central  office  for  all 
bases  to  contact  --  past  performance  seems  to  hold  more 
weight  these  days!). 

We  ( ESD )  are  currently  experimenting  with  a 
"streamlined"  source  selection.  However,  just  the  technical 
area  has  been  shortened  --  cost  takes  the  same  amount  of 
time,  and  also  the  reports  (SSEB  and  SSAC )  are  still  very 
lengthy.  Maybe  if  these  areas  were  streamlined  too,  source 
selection  would  be  faster. 

Capt:  In  general,  the  source  selection  process  is  an 

abyss  that  is  a  mystery  until  you  are  "fortunate"  enough  to 
have  been  a  participant  on  one.  Then  you  learn  everything 
you  need  to  know  about  it  the  hard  way.  You  stumble  and 
bumble  your  way  through  and  eventually  become  an  expert. 

What  is  needed  is  a  permanent  source  selection  staff  agency 
that  goes  through  all  the  wickets,  having  only  technical 
experts  assigned  for  each  peculiar  effort.  After  all,  the 
rest  is  merely  administrative  and  should  not  require  novices 
learning  this  process  all  the  time. 
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